A
B
THEDEFI
NI
TI
VEGUI
DETO

MUL
TI
VARI
ATEANDABTESTI
NG
www.
i
nv
espcr
o.
com


CONTENT
I
nt
r
oduct
i
on

01

Whati
sA/
Bt
est
i
ng(
spl
i
tt
est
i
ng)
?

03

Whati
sMul
t
i
v
ar
i
at
eTest
i
ng?

11


TheDi
f
f
er
enceBet
weenA/
BTest
i
ngandMul
t
i
v
ar
i
at
eTest
i
ng

18

Compar
i
ngMul
t
i
v
ar
i
at

e&ABTest
i
ngTool
s(
updat
edf
or2018)

23

ThePr
ocessofCr
eat
i
ngaSuccessf
ul
A/
BTest

29

WhatEl
ement
sShoul
dYouI
ncl
udei
nanA/
BTest
?


34

Mul
t
i
v
ar
i
at
eandA/
BTest
i
ngBestPr
act
i
ces

54

16mi
st
ak
est
hatwi
l
l
k
i
l

l
y
ourA/
Bt
est
i
ng (
andwhaty
oucandoaboutt
hem)

61


I
NTRODUCTI
ON
Mostwebsi
t
escont
i
nuet
ost
r
uggl
ewi
t
hi
ncr
easi

ngt
hei
ronl
i
neconv
er
si
onr
at
es.
Onav
er
age,
anecommer
cest
or
econv
er
t
sar
ound3%ofi
t
st
r
af
fici
nt
ocust
omer
s.

Ther
emai
ni
ng97%
oft
hev
i
si
t
or
sl
eav
et
hewebsi
t
ewi
t
houtev
erpl
aci
nganor
der
.
Ot
hert
y
pesofwebsi
t
esst
r

uggl
ewi
t
hconv
er
si
onaswel
l
.
Onav
er
age,
al
eadgener
at
i
onorsubscr
i
pt
i
on
websi
t
esconv
er
tar
ound15%ofi
t
st
r

af
fici
nt
ocust
omer
s.
Ther
emai
ni
ng85%l
eav
et
hewebsi
t
ewi
t
hout
ev
erpl
aci
nganor
der
.
I
fy
ouar
el
ook
i
ngt

oi
ncr
easey
ourwebsi
t
econv
er
si
onr
at
e,
y
ouwi
l
l
hav
et
oconductABandmul
t
i
v
ar
i
at
e
t
est
i
ng.
Thepr

obl
em i
st
hatmostt
est
i
ngpr
ogr
amsst
ar
twi
t
houtanypl
anni
ngandt
husf
ai
l
wi
t
ht
hefir
st
6mont
hs.
Wek
nowt
hi
sfir
sthandbecausewehav

ebeenhel
pi
ngonl
i
necompani
esconductt
est
i
ngf
orov
er12
y
ear
s.
Ourgoal
f
r
om t
hi
sgui
det
ohel
py
ouwi
t
hagoodi
nt
r
oduct
i

ont
omul
t
i
v
ar
i
at
eandABt
est
i
ng.
We
wi
l
l
cont
i
nueupdat
i
ngt
hedi
f
f
er
entchapt
er
si
nt
hebookonr

egul
arbasi
s,
soi
fy
ouhav
ecomment
sor
f
eedback
,
pl
easel
et
’
sk
now(
@i
nv
esp)
.

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI

ATEANDABTESTI
NG

01


C H A P TER 1
WHATI
SA/
BTESTI
NG(
SPLI
TTESTI
NG)
?
WRI
TTENBY:KHALI
DSALEH


C H A P TER 1
WHATI
SA/
BTESTI
NG(
SPLI
TTESTI
NG)
?
I

fv
i
si
t
or
sar
enotconv
er
t
i
ngony
ourwebsi
t
e,
t
henobv
i
ousl
y
,
t
her
ei
sapr
obl
em t
hati
sst
oppi
ngt

hem.
Youcangoaheadandasky
ourdesi
gnt
eam t
ocr
eat
enewdesi
gns,
butt
hequest
i
onr
emai
ns:
howdoy
ouk
nowt
hatt
henewdesi
gnswi
l
l
conv
er
tmor
ev
i
si
t

or
scompar
edt
ot
heor
i
gi
nal
desi
gn?
Thati
swher
eABt
est
i
ngcomesi
nhandy
.
A/
Bt
est
i
ng(
somet
i
mesr
ef
er
r
edt

oasspl
i
tt
est
i
ng)i
st
hepr
ocessoft
est
i
ngmul
t
i
pl
enewdesi
gns
ofawebpageagai
nstt
heor
i
gi
nal
desi
gnoft
hatpagewi
t
ht
hegoal
ofdet

er
mi
ni
ngwhi
chdesi
gngener
at
esmor
econv
er
si
ons.
Theor
i
gi
nal
desi
gnofapagei
susual
l
yr
ef
er
r
edt
oast
hecont
r
ol
.

Thenewdesi
gnsoft
hepagear
eusual
l
yr
ef
er
r
edt
oast
he“
v
ar
i
at
i
ons”
,
“
chal
l
enger
s”or“
r
eci
pes.
”Thepr
ocessoft
est

i
ngwhi
chpagedesi
gngener
at
esmor
econv
er
si
onsi
st
y
pi
cal
l
yr
ef
er
r
edt
oasa“
t
est
”oran
“
ex
per
i
ment
.

”
A“
conv
er
si
on”wi
l
l
v
ar
ybasedony
ourwebsi
t
eandt
hepagey
ouar
et
est
i
ng.
Foranecommer
cewebsi
t
e,
aconv
er
si
oncoul
dbeav
i

si
t
or
pl
aci
nganor
der
.
ForaSaaSwebsi
t
e,
aconv
er
si
oncoul
dbeav
i
si
t
orsubscr
i
bi
ngt
ot
heser
v
i
ce.
Foral
eadgener

at
i
onwebsi
t
e,
aconv
er
si
on
coul
dbeav
i
si
t
orfil
l
i
ngoutacont
actf
or
m.

1STEXAMPLE
Thehomepageonanecommer
cewebsi
t
er
ecei
v
es100,

000v
i
si
t
or
samont
h.
Todet
er
mi
nei
ft
her
ei
sawayt
oi
ncr
ease
conv
er
si
ons,
t
hedesi
gnt
eam cr
eat
esonenewdesi
gnf
ort

hehomepage.
ABt
est
i
ngsof
t
war
ei
st
henusedt
or
andoml
yspl
i
tt
hehomepagev
i
si
t
or
sbet
weent
hecont
r
ol
andt
henewchal
l
enger
.

So,
50,
000
v
i
si
t
or
sar
edi
r
ect
edt
ot
hecont
r
ol
and50,
000v
i
si
t
or
sar
edi
r
ect
edt
ot
hechal

l
enger
.
Si
ncewear
et
est
i
ngwhi
chdesi
gngener
at
es
mor
eor
der
s(
conv
er
si
ons)
,
t
henweuset
heABt
est
i
ngsof
t
war

et
ot
r
ackt
henumberofconv
er
si
onseachdesi
gngener
at
es.
The
A/
Bt
est
i
ngsof
t
war
ewi
l
l
t
hendet
er
mi
net
hewi
nni
ngdesi

gnbasedont
henumberofconv
er
si
ons.

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

03


2NDEXAMPLE
Thehomepagef
orabl
ogr
ecei
v
es3,
000v
i
si

t
or
samont
h.
Thepr
i
mar
yconv
er
si
ongoal
f
ort
hehomepagei
st
ogetav
i
si
t
ort
o
subscr
i
bet
ot
heemai
l
l
i
stoft

hebl
og.
Thedesi
gnercr
eat
esanewdesi
gnf
ort
hebl
oghomepagewhi
chhi
ghl
i
ght
st
he
subscr
i
pt
i
onbox
.
Thespl
i
tt
est
i
ngsof
t
war

ei
susedt
osend1,
500v
i
si
t
or
st
ot
heor
i
gi
nal
pagedesi
gn(
cont
r
ol
)
,
andt
het
est
i
ngsof
t
war
esends
1,

500v
i
si
t
or
st
ot
henewdesi
gn(
chal
l
enger
)
.
Thet
est
i
ngsof
t
war
et
r
ack
st
henumberofsubscr
i
ber
s(
conv
er

si
ons)eachdesi
gn
gener
at
es.
A2015sur
v
eybyEconsul
t
ancyshowedt
hat58%ofi
t
sr
espondent
sar
econduct
i
ngA/
Bt
est
i
ng:

Buthowsuccessf
ul
i
sABt
est
i

ngi
nhel
pi
ngcompani
esi
ncr
easet
hei
rconv
er
si
onr
at
es?
A2017sur
v
eybyOpt
i
mi
z
el
yshowst
hatonl
y25%ofal
l
A/
Bt
est
spr
oducesi

gni
ficant
l
yposi
t
i
v
er
esul
t
s.
Vi
sual
Websi
t
eOpt
i
mi
z
err
epor
t
s
t
hatonl
y12%ofal
l
A/
Bt
estpr

oducesi
gni
ficant
l
yposi
t
i
v
er
esul
t
s.
Fi
nal
l
y
,
dat
af
r
om Googl
eshowst
hatonl
y10%ofal
l
A/
Bt
estpr
oduce
si

gni
ficant
l
yposi
t
i
v
er
esul
t
s.

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

04


CaseSt
udy
:WhatNet
f

l
i
xl
ear
nedf
r
om ABt
est
i
ng?

Backgr
ound
46%ofsur
v
ey
edNet
f
l
i
xv
i
si
t
or
scompl
ai
nedt
hatt
hewebsi

t
edoesnotal
l
owt
hem t
ov
i
ewmov
i
et
i
t
l
esbef
or
esi
gni
ngupf
ort
heser
v
i
ce.
So,
Net
f
l
i
xdeci
dedt

or
unanA/
Bt
estont
hei
rr
egi
st
r
at
i
onpr
ocesst
oseei
far
edesi
gnedr
egi
st
r
at
i
onpr
ocesswi
l
l
hel
pi
ncr
easesubscr

i
pt
i
ons.
Cr
eat
i
nganABt
est
Thenewdesi
gndi
spl
ay
edmov
i
et
i
t
l
est
ov
i
si
t
or
spr
i
ort
or
egi

st
r
at
i
on.
TheNet
f
l
i
xt
eam want
edt
ofindouti
ft
henewdesi
gnwi
t
hmov
i
et
i
t
l
es
woul
dgener
at
emor
er
egi

st
r
at
i
onscompar
edt
ot
heor
i
gi
nal
desi
gnwi
t
houtt
het
i
t
l
es.
Thi
swasanal
y
z
edbyr
unni
nganA/
Bt
estbet
ween

t
henewdesi
gnsagai
nstt
heor
i
gi
nal
desi
gn.
Thet
esthy
pot
hesi
swasst
r
ai
ght
f
or
war
d:
Al
l
owi
ngv
i
si
t
or

st
ov
i
ewav
ai
l
abl
emov
i
et
i
t
l
esbef
or
er
egi
st
er
i
ngwi
l
l
i
ncr
easet
henumberofnew
si
gnups.
I

nt
hespl
i
tt
est
,
t
het
eam i
nt
r
oducedfiv
edi
f
f
er
entchal
l
enger
sagai
nstt
heor
i
gi
nal
desi
gn.
Thet
eam t
henr

ant
het
estt
oseet
hei
mpact
.
Whatwer
et
her
esul
t
s?
Resul
t
soft
heABt
estandanal
ysi
s
Theor
i
gi
nal
desi
gnconsi
st
ent
l
ybeatal

l
chal
l
enger
s.
Ther
eal
anal
y
si
shappensaf
t
ery
ouconcl
udeanA/
Bt
est
.
Whydi
dt
heor
i
gi
nal
desi
gn
beatal
l
newdesi
gnsal

t
hough46%ofv
i
si
t
or
ssai
dt
hatseei
ngwhatt
i
t
l
esNet
f
l
i
xcar
r
i
eswi
l
l
per
suadet
hem t
osi
gnupf
ort
heser

v
i
ce?
Thet
eam atNet
f
l
i
xgav
et
hr
eedi
f
f
er
entr
easonsofwhyt
heor
i
gi
nal
desi
gnbeatal
l
t
hechal
l
enger
s:
1.Net

f
l
i
xi
sal
laboutt
heexper
i
ence:t
hemor
euser
si
nt
er
act
edwi
t
ht
hewebsi
t
e,
t
hemor
et
heyl
ov
et
heex
per
i

ence.
So,
Net
f
l
i
xi
smor
e
t
hanj
ustbr
owsi
ng.
2.Si
mpl
i
f
ychoi
ce:t
heor
i
gi
nal
desi
gn(
t
hecont
r
ol

)showeduser
soneopt
i
on:
si
gnupf
ort
heser
v
i
ce.
Thenewdesi
gnsof
f
er
edv
i
si
t
or
s
mul
t
i
pl
eopt
i
ons(
mul
t

i
pl
emov
i
es)
.
Thi
scompl
i
cat
edt
hechoi
cewhi
chv
i
si
t
or
shadt
omak
e.
Mor
echoi
cesr
esul
t
edi
nf
ewer
conv

er
si
ons.
3.User
sdonotal
waysknowwhatt
heywant
:TheNet
f
l
i
xt
eam ar
guedt
hatt
estr
esul
t
spoi
ntt
ot
hef
actt
hatuser
sdonotal
way
sk
now
whatt
heywant

.

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

05


Whi
l
et
hesemi
ghtbev
al
i
dex
pl
anat
i
ons,
especi
al

l
yt
hesecondpoi
nt
,
wewoul
dar
guet
hatt
her
ei
sanot
herr
easonal
t
oget
her
.
Coul
di
tbet
hatv
i
si
t
or
sfinal
l
yseeal
l

t
hemov
i
eopt
i
onswhi
chNet
f
l
i
xof
f
er
sandt
heydonotfindt
hemov
i
esel
ect
i
onconv
i
nci
ng,
sot
hey
deci
det
owal
kaway

?I
ft
hati
st
hecase,
i
st
hepr
obl
em wi
t
ht
henewdesi
gnsori
si
tapr
obl
em i
nt
hemov
i
esel
ect
i
onwhi
cht
hesi
t
eof
f

er
s?

Howdoest
heA/
Bt
est
i
ngsof
t
war
edet
er
mi
net
hewi
nni
ngdesi
gn?

Ati
t
scor
e,
ABt
est
i
ngsof
t
war

et
r
ack
st
henumberofv
i
si
t
or
scomi
ngt
oeachdesi
gni
nanex
per
i
mentandt
henumberofconv
er
si
onseach
desi
gngener
at
es.
Sophi
st
i
cat
edA/

Bt
est
i
ngsof
t
war
et
r
ack
smuchmor
edat
af
oreachv
ar
i
at
i
on.
Asanex
ampl
e,
Fi
gPi
i
t
r
ack
s:
Conver
si

ons
Pagevi
ews

Vi
si
t
or
s

Bouncer
at
e

Revenuepervi
si
t

Exi
t
Revenue

Sour
ceoft
r
af
fic
Medi
um oft
r

af
fic

Thespl
i
tt
est
i
ngsof
t
war
eusesdi
f
f
er
entst
at
i
st
i
cal
modest
odet
er
mi
neawi
nneri
nat
est
.

Thet
wopopul
armet
hodsf
ordet
er
mi
ni
nga
wi
nnerar
eFr
equent
i
standBay
esi
anmodel
s.
Thespl
i
tt
est
i
ngsof
t
war
et
r
ack
sconv

er
si
onr
at
esf
oreachdesi
gn.
Howev
er
,
decl
ar
i
ngawi
nneri
naspl
i
tt
estr
equi
r
esmor
et
hangener
at
i
ng
asmal
l
i

ncr
easei
nconv
er
si
onr
at
escompar
edt
ot
hecont
r
ol
.
TheFr
equent
i
stmodel
usest
womai
nf
act
or
st
odet
er
mi
net
hewi
nni

ngdesi
gn:
Theconver
si
onr
at
ef
oreachdesi
gn:t
hi
snumberi
sdet
er
mi
nedbydi
v
i
di
ngt
henumberofconv
er
si
onsf
oradesi
gnbyt
heuni
que
v
i
si

t
or
sf
ort
hatdesi
gn.
Theconfidencel
evelf
oreachdesi
gn:ast
at
i
st
i
cal
t
er
mi
ndi
cat
i
ngt
hecer
t
ai
nt
yt
haty
ourt
estt

hatwi
l
l
pr
oducet
hesamer
esul
ti
ft
he
sameex
per
i
menti
sconduct
edacr
ossmanysepar
at
edat
aset
si
ndi
f
f
er
entex
per
i
ment
s.

Thi
nkofconfidencel
ev
el
ast
hepr
obabi
l
i
t
yofhav
i
ngar
esul
t
.
So,
i
fachal
l
engerpr
oducesa20%i
ncr
easei
nconv
er
si
onswi
t
ha95%

confidence,
t
heny
ouassumet
haty
ouhav
eanex
cel
l
entpr
obabi
l
i
t
yofget
t
i
ngt
hesamer
esul
twhensel
ect
i
ngt
hatchal
l
engerasy
our
def
aul

tdesi
gn.
I
tal
soi
ndi
cat
est
haty
ouhav
ea5%chancet
haty
ourt
estr
esul
t
swer
eduet
or
andom chance,
anda5%possi
bi
l
i
t
yt
haty
ou
f
oundawr

ongwi
nner
.
TheBay
esi
anmodel
usest
womai
nf
act
or
st
odet
er
mi
net
hewi
nni
ngdesi
gn:
Theconver
si
onr
at
ef
oreachdesi
gn:asdefinedabov
e.
Hi
st

or
i
calper
f
or
mance:t
hesuccessr
at
eofpr
ev
i
ousl
yr
anA/
Bex
per
i
ment
sr
anont
hewebpage.

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI

ATEANDABTESTI
NG

06


“

“

Leoni
dPek
el
i
s,
Opt
i
mi
z
el
y
’
sfir
sti
nhousest
at
i
st
i
ci

an,
ex
pl
ai
nst
hi
sbysay
i
ng

Bay
esi
anst
at
i
st
i
cst
akeamor
ebot
t
omupappr
oacht
odat
aanal
ysi
s.Thi
s

meanst

hatpastknowl
edgeofsi
mi
l
arexper
i
ment
si
sencodedi
nt
oast
at
i
st
i
cal
devi
ceknownasapr
i
or
,
andt
hi
spr
i
ori
scombi
nedwi
t
hcur

r
entexper
i
ment
dat
at
omakeaconcl
usi
onont
het
estathand.

Wet
y
pi
cal
l
yr
el
yonmul
t
i
pl
emet
r
i
cswhendet
er
mi
ni

ngawi
nni
ngdesi
gnf
orat
est
.
Mostofourecommer
cecl
i
ent
suseacombi
nat
i
onof
conv
er
si
onr
at
esandr
ev
enueperv
i
si
tt
odet
er
mi
neafinal

wi
nneri
nanex
per
i
ment
.
Sel
ect
i
ngwhi
chmet
r
i
cswi
l
l
dependony
ourspeci
ficsi
t
uat
i
on.
Howev
er
,
i
ti
scr

uci
al
t
ochoosemet
r
i
cst
hathav
eani
mpactony
ourbot
t
om
l
i
ne.
Opt
i
mi
z
i
ngf
oral
owerbounceorex
i
tr
at
eswi
l
l

hav
el
i
t
t
l
edi
r
ectandmeasur
abl
edol
l
arv
al
uet
omostbusi
nesses.
Thet
eam atBi
ngwast
r
y
i
ngt
ofindawayt
oi
ncr
easet
her
ev

enuewhi
cht
hesi
t
egener
at
esf
r
om ads.
Todoso,
t
heyi
nt
r
oducedanew
desi
gnt
hatemphasi
z
edhowsear
chadsar
edi
spl
ay
ed.Thet
eam t
est
edt
henewdesi
gnv

s.
t
heol
ddesi
gn.
Thespl
i
tt
estr
esul
t
sshoweda
30%i
ncr
easei
nr
ev
enueperv
i
si
t
.
Thi
s,
howev
er
,
wasduet
oabugi
nt

hei
rmai
nsear
chr
esul
t
sal
gor
i
t
hm i
nt
henewdesi
gn.
Thi
sbugshowedv
i
si
t
or
spoorsear
chr
esul
t
s.
And
asar
esul
t
,

v
i
si
t
or
swer
ef
r
ust
r
at
edandwer
ecl
i
ck
i
ngonads.
Whi
l
et
henewdesi
gngener
at
edahi
gherr
ev
enueperv
i
si
t

,
t
hi
swasnotagoodl
ongt
er
m st
r
at
egy
.Thet
eam deci
dedt
ost
i
ckt
ot
heol
d
desi
gni
nst
ead.

Assi
gni
ngwei
ght
edt
r

af
fict
odi
f
f
er
entvar
i
at
i
ons
MostABt
est
i
ngsof
t
war
eaut
omat
i
cal
l
ydi
v
i
desv
i
si
t
or

sequal
l
ybet
weendi
f
f
er
entv
ar
i
at
i
ons.
Ther
ear
ehowev
eri
nst
anceswher
ey
ouneedt
oassi
gndi
f
f
er
entwei
ght
st
odi

f
f
er
entv
ar
i
at
i
ons.
Forex
ampl
e,
l
et
’
st
ak
eanex
per
i
mentt
hathasanor
i
gi
nal
desi
gnandt
wochal
l
enger

si
ni
t
.
Thet
est
i
ngt
eam mi
ghtwantt
oassi
gn50%of
t
hev
i
si
t
or
st
ot
heor
i
gi
nal
desi
gnandspl
i
tt
her
emai

ni
ng50%bet
weenv
ar
i
at
i
onsoneandt
wo.

Shoul
dyour
unABt
est
i
ngon100%ofyourvi
si
t
or
s?
SomeConv
er
si
onopt
i
mi
z
at
i
onex

per
t
sdebat
et
hi
squest
i
onatgr
eatl
engt
hs.
Look
i
ngaty
ouranal
y
t
i
cs,
y
oucant
y
pi
cal
l
ynot
i
cet
hatdi
f

f
er
entv
i
si
t
orsegment
si
nt
er
actdi
f
f
er
ent
l
ywi
t
hy
ourwebsi
t
e.
Ret
ur
ni
ngv
i
si
t
or

s
(
t
hosewhov
i
si
t
edsi
t
epr
ev
i
ousl
y
)gener
al
l
yar
emor
eengagedwi
t
ht
hewebsi
t
ecompar
edt
onewv
i
si
t

or
s.

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

07


Whenl
aunchi
nganewABt
est
,
y
ouwi
l
l
not
i
cet
hati

nmanyi
nst
ances:
Newv
i
si
t
or
sr
eacti
nabet
t
erwaywi
t
hy
ourex
per
i
mentchal
l
enger
s.
Ret
ur
ni
ngv
i
si
t
or

s,
whoar
eusedt
oy
ourcur
r
entdesi
gn,
r
eactnegat
i
v
el
yt
oy
ournewdesi
gns.
Thef
actt
hatnewv
i
si
t
or
sconv
er
tathi
gherr
at
eswi

t
hnewdesi
gnscompar
edt
or
et
ur
ni
ngv
i
si
t
or
si
sat
t
r
i
but
edt
ot
het
heor
yof
moment
um behav
i
or
.
I

fy
ouy
ourwebsi
t
eget
sal
ar
genumberofv
i
si
t
or
s,
wer
ecommendt
haty
oul
aunchnewt
est
sf
oronl
ynewv
i
si
t
or
sandobser
v
i
nghowt

hey
r
eactt
oi
t
.
Af
t
ert
hat
,
y
oucanst
ar
tt
het
estf
orr
et
ur
ni
ngv
i
si
t
or
sandcompar
et
hei
rr

eact
i
onst
ot
henewdesi
gnsi
nt
r
oducedi
nt
he
ex
per
i
ment
.

Hol
dbackspl
i
tt
est
s
100%

50%
25%
5%

3


3

3

Unt
i
lwi
nneri
sdecl
ar
ed

Wet
y
pi
cal
l
yr
ecommendr
unni
nghol
dbackspl
i
tt
est
sf
orl
ar
gerwebsi

t
est
hatr
ecei
v
et
housandsofconv
er
si
onspermont
h.
I
nt
heset
y
pes
oft
est
s,
y
oul
auncht
het
est
st
oasmal
l
per
cent
ageofy

oursi
t
ev
i
si
t
or
s.
Forex
ampl
e,
y
oust
ar
twi
t
hl
aunchi
ngt
het
estt
o10%ofy
our
v
i
si
t
or
s.
I

ft
her
esul
t
sar
eencour
agi
ng,
t
heny
ouex
pandt
het
estt
o25%,
50%,
and100%ofy
ourwebsi
t
ev
i
si
t
or
s.
Ther
ear
esev
er
al

adv
ant
agest
or
unni
nghol
dbackA/
Bt
est
s:
Di
scoveranyt
est
i
ngbugs:Asy
oul
aunchanABt
est
,
y
ourdesi
gnsmi
ghthav
ebugsi
nt
hem.
Byr
unni
ngt
het

estonasmal
l
per
cent
age
ofy
ourv
i
si
t
or
s,
onl
yt
hatt
i
nysegmentoft
hev
i
si
t
or
swi
l
l
seet
heer
r
or
si

nt
henewdesi
gns.
Thatwi
l
l
gi
v
ey
out
heoppor
t
uni
t
yt
ofixt
hese
bugsbef
or
er
ol
l
i
ngoutt
het
estt
o100%ofy
ourv
i
si

t
or
s.
Reducer
evenuer
i
sk:byr
unni
ngt
het
estonasmal
l
per
cent
ageofv
i
si
t
or
s,
y
our
educet
her
i
skofhav
i
ngoneofy
our
st

estv
ar
i
at
i
on
causi
ngasi
gni
ficantdr
opi
nr
ev
enue.
I
fy
ouchooset
or
unhol
dbackA/
Bt
est
s,
mak
esur
et
haty
oust
ar
tanewt

esteacht
i
mey
ouchanget
het
r
af
fical
l
ocat
i
ongoi
ngt
hr
ought
he
ex
per
i
mentt
oav
oi
danyst
at
i
st
i
cal
pr
obl

emswi
t
hr
esul
t
s.

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

08


Howmanyvar
i
at
i
onsshoul
dyoui
ncl
udei
nat

est
?
Ther
ei
sal
otofmat
ht
hatgoesi
nt
odet
er
mi
ni
nghowmanyv
ar
i
at
i
onsshoul
dbei
ncl
udedi
nanA/
Bt
est
.
Thef
ol
l
owi

ngar
egener
al
gui
del
i
nesy
oucanappl
y
,
howev
er
,
mor
edet
ai
l
swi
l
l
becov
er
edi
nal
at
ersect
i
on:
Cal
cul

at
et
hemont
hl
ynumberofconv
er
si
onsgener
at
edbyt
hepar
t
i
cul
arpagey
oupl
ant
ot
est
:
ont
heconser
v
at
i
v
esi
de,di
v
i

det
het
ot
al
mont
hl
yconv
er
si
onsgener
at
edbyt
hepageby500andsubt
r
actone
ont
heaggr
essi
v
esi
de,di
v
i
det
het
ot
al
mont
hl
yconv

er
si
onsgener
at
edbyt
hepageby200andsubt
r
actone

“

I
fy
ouhav
el
esst
han200conv
er
si
onsamont
h,
y
ourwebsi
t
ei
snotr
eadyf
orA/
Bt
est

i
ng.
Focusondr
i
v
i
ngmor
ev
i
si
t
or
st
oy
ourwebsi
t
e.

“

Yourwebsi
t
egener
at
es1,
000conv
er
si
onspermont
h:


Ont
heconser
v
at
i
v
esi
de,
anA/
Bt
estcani
ncl
udeonechal
l
engeragai
nstt
he
or
i
gi
nal(
1000/500–1)

Ont
heaggr
essi
v
esi
de,

anA/
Bt
estcani
ncl
udef
ourchal
l
enger
sagai
nstt
he
or
i
gi
nal(
1000/200–1)

Agai
n,
t
hi
si
sasi
mpl
i
ficat
i
onoft
hecal
cul

at
i
on,
buti
twi
l
lgi
v
ey
ouagood
st
ar
t
i
ngpoi
nt
.

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

09



C H A P TER 2
WHATI
SMUL
TI
VARI
ATETESTI
NG?
WRI
TTENBY:KHALI
DSALEH


C H A P TER 2
WHATI
SMUL
TI
VARI
ATETESTI
NG?

Edi
t
ornot
e:
Pl
easenot
et
hatweusebut

t
oncol
or
s,
f
ont
s,
headl
i
nesasa
wayt
ocl
ar
i
f
yt
heconceptoft
est
i
ng.
Succesf
ul
ABand
mul
t
i
v
ar
i
at

et
est
swi
l
l
i
ncl
udemor
esophi
st
i
cat
edchangest
o
y
ourpage.
Mul
t
i
v
ar
i
at
eTest
i
ngorMVTt
est
i
ngi
st

hepr
ocessoft
est
i
ng
mul
t
i
pl
ev
ar
i
at
i
onsofmul
t
i
pl
eel
ement
sonawebpagewi
t
h
t
hegoal
ofdet
er
mi
ni
ngt

hebestcombi
nat
i
onofdi
f
f
er
ent
el
ement
sont
hepaget
oi
ncr
easeconv
er
si
ons.
Byusi
ngMVTt
est
i
ngsof
t
war
e,
y
oucant
estdi
f

f
er
ent
v
ar
i
at
i
onsofanyel
ementony
ourpage(
headl
i
nes,
i
mages,
but
t
ons,
et
c.
)t
omeasur
et
hei
ri
mpactony
ourconv
er
si

on
r
at
es.
Thef
ol
l
owi
ngi
magedi
spl
ay
sanex
ampl
eofhowMVT
t
est
i
ngsof
t
war
ewor
k
s.
I
nt
hi
sex
ampl
e,

t
hesof
t
war
et
est
sdi
f
f
er
entv
ar
i
at
i
onsoft
hepageheadl
i
ne,
i
mage,
andt
he“
cal
l
t
oact
i
on”but
t

on:
Theor
i
gi
nal
headl
i
nei
st
est
edagai
nstt
hr
eeot
herpossi
bl
eheadl
i
nes,
f
orat
ot
al
off
ourpossi
bl
eheadl
i
nesont
hepage

Theor
i
gi
nal
i
magei
st
est
edagai
nstt
woot
herpossi
bl
ephot
os,
f
orat
ot
al
oft
hr
eepossi
bl
epi
ct
ur
esont
hepage
Thr
eedi

f
f
er
entbut
t
onsar
et
est
edagai
nstt
heor
i
gi
nal
but
t
onont
hepage,
f
orat
ot
al
off
ourpossi
bl
ebut
t
onsont
hepage
Asav

i
si
t
orar
r
i
v
esatapage,
t
hesof
t
war
epi
ck
soneoft
hef
ourheadl
i
nes,
oneoft
het
hr
eei
mages,
andoneoft
hef
ourbut
t
onst
odi

spl
ay
.

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

11


Yourt
eam doesnothav
et
ocr
eat
eal
l
oft
he48desi
gns;
t
hesof

t
war
ewi
l
l
swapt
hedi
f
f
er
entv
ar
i
at
i
onsandcr
eat
et
hedesi
gnsaut
omat
i
cal
l
y
andcr
eat
eal
l
48possi

bl
ev
ar
i
at
i
ons.
Thef
ol
l
owi
ngi
mageshowsf
ouroft
he48possi
bl
edesi
gnst
het
est
i
ngsof
t
war
ecangener
at
e.

Or
i

gi
nalHeadl
i
ne

Or
i
gi
nalHeadl
i
ne

Or
i
gi
nalI
mage

2ndI
mage

Or
i
gi
nalBut
t
on

Or
i

gi
nalBut
t
on

ORI
GI
NAL

DESI
GN1

Or
i
gi
nalHeadl
i
ne

2ndHeadl
i
ne

3ndI
mage

2ndI
mage

Or

i
gi
nalBut
t
on

Or
i
gi
nalBut
t
on

DESI
GN2

DESI
GN3

Thet
ot
al
numberoft
est
i
ngv
ar
i
at
i

ons(
al
socal
l
edchal
l
enger
s)dependsont
henumberofel
ement
sy
ouwi
l
l
t
estonapage(
headl
i
ne,
i
mage,
but
t
ons,
et
c.
)andt
henumberofv
ar
i

at
i
onsy
ouwi
l
l
bet
est
i
ngf
oreachoft
heseel
ement
s.
Youcancal
cul
at
et
het
ot
al
numberofchal
l
enger
si
namul
t
i
v
ar

i
at
et
estmul
t
i
pl
y
i
ngt
henumberofdi
f
f
er
entv
ar
i
at
i
onsofeachoft
he
el
ement
s.
Forawebpagei
nwhi
chwewi
l
l
bet

est
i
ng(
N)numberofel
ement
s,
wecal
cul
at
e:
Tot
al
numberofpagev
ar
i
at
i
ons=Numberofv
ar
i
at
i
onsof1stel
ementxNumberofv
ar
i
at
i
onsoft
he2ndel

ementxNumberofv
ar
i
at
i
ons
oft
he3r
del
ementx…xNumberofv
ar
i
at
i
onsoft
heNt
hel
ement
Thenumberofpagev
ar
i
at
i
onscangr
owv
er
yf
ast
.
Somet

est
i
ngsof
t
war
eal
l
owsy
out
ot
ensoft
housands(
somet
i
mesmi
l
l
i
ons)of
v
ar
i
at
i
onsofasi
ngl
epage.

THEDEFI
NI

TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

12


MVTTestExampl
es
Let
’
st
aket
hepr
oductpagef
r
om appl
e.
com asanexampl
e:

Ont
hi
spage,
y

oucant
est
:
di
f
f
er
entv
ar
i
at
i
onsoft
heheadl
i
ne

di
f
f
er
entpr
oducti
mages

CTAcol
or
s

di

spl
ayi
ngt
woMacBookmodel
sperl
i
ne
(
cur
r
ent
l
y
,
eachmodel
t
ak
esal
i
ne)

di
f
f
er
entpr
i
ci
ng


CTAt
ext

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

13


Let
’
st
akeanexampl
ef
r
om sal
esf
or
ce.
com:

Ont

hi
spage,
y
oucant
est
:
di
f
f
er
entv
ar
i
at
i
onsoft
heheadl
i
ne

di
f
f
er
enther
oi
mages

di
spl

ayi
ngt
hesi
denavi
gat
i
onornotshowi
ngi
t

CTAcol
or
s

CTAt
ext

Howdot
ocr
eat
easuccessf
ulmul
t
i
var
i
at
et
est
?

Mul
t
i
v
ar
i
at
et
est
i
ngsof
t
war
eal
l
owsmar
k
et
er
st
ocr
eat
eandst
ar
tsi
mpl
et
est
si
naf

ewhour
s.
Butt
hati
st
heeasypar
t
!
Manycompani
es
ul
t
i
mat
el
yf
ai
l
whendesi
gni
ngsuccessf
ul
t
estscenar
i
os,
assessi
ngr
esul
t

s,
andcr
eat
i
ngmeani
ngf
ul
f
ol
l
owupt
est
s.
Poor
l
ydesi
gnedex
per
i
ment
scant
ak
ey
ear
st
oconcl
ude.
Ev
enwor
se,

t
heymi
ghtnotpr
ov
i
deaccur
at
ei
nsi
ght
si
nt
owhatel
ement
sconv
er
t
mor
ev
i
si
t
or
si
nt
ocust
omer
s.
I
magi

neacasewher
ey
oupl
ant
ot
estdi
f
f
er
entheadl
i
nesonapage.
Youst
ar
tbycomi
ngupwi
t
ht
endi
f
f
er
entpossi
bl
ev
ar
i
at
i
onst

ot
he
headl
i
nes.
Whi
choft
heset
enpossi
bl
eheadl
i
nesshoul
dy
out
estagai
nsty
ouror
i
gi
nal
headl
i
ne?Whynott
estal
l
oft
hem?Whynott
est
v

ar
i
at
i
onsofi
mages,
but
t
ons,
andl
ay
out
s?
Youwi
l
l
mostl
i
k
el
yfindy
our
sel
fr
el
y
i
ngonguesswor
kt
odet

er
mi
newhi
chv
er
si
onst
oi
ncl
udei
nt
het
est
.
Thesamel
ogi
c,
ofcour
se,
appl
i
es
t
oal
l
el
ement
sy
ouwantt
ot

estonapage.
Wi
t
houtbei
ngj
udi
ci
ouswi
t
ht
estscenar
i
os,
y
oumi
ghtendupat
t
empt
i
ngt
ot
estmi
l
l
i
onsofcombi
nat
i
ons.


THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

14


Test
i
ngi
sanessent
i
al
componentofanyconv
er
si
onopt
i
mi
z
at
i
onpr

oj
ect
.
Howev
er
,
i
tshoul
dnotbet
heonl
ycomponent
.
Test
i
ngshoul
d
onl
yt
ak
epl
aceaf
t
ert
heconcl
usi
onofot
herequal
l
ycr
i

t
i
cal
opt
i
mi
z
at
i
onst
ages,
suchasper
sonadev
el
opment
,
v
oi
ceofcust
omerr
esear
ch
(
i
ncl
udi
ngpol
l
sandsur
v

ey
s)
,
heur
i
st
i
cev
al
uat
i
on,
usabi
l
i
t
yt
est
i
ng,
si
t
eanal
y
si
s,
anddesi
gnandcopycr
eat
i

on.
Eachoft
heseel
ement
s
pr
ov
i
desabui
l
di
ngbl
ockt
owar
dsahi
ghl
yopt
i
mi
z
edwebsi
t
et
hatconv
er
t
sv
i
si
t

or
si
nt
ocust
omer
s.
Tocr
eat
easuccessf
ul
t
est
,
y
oumustgot
hr
ought
hef
ol
l
owi
ngst
eps:
Ev
al
uat
et
hepage,
l
ook

i
ngf
orpossi
bl
epr
obl
emsi
ni
t
Pr
i
or
i
t
i
z
et
hei
ssuesi
dent
i
fiedont
hepagei
nt
er
msoft
hei
ri
mpactony
ourconv

er
si
onr
at
e
Cr
eat
eahy
pot
hesi
sofhowt
ofixsomeoft
het
opi
ssuesont
hepageandt
heef
f
ecty
ourfixwi
l
l
hav
eony
ourconv
er
si
onr
at
e

Asser
tt
hev
al
i
di
t
yofy
ourhy
pot
hesi
st
hr
oughmul
t
i
v
ar
i
at
eorABt
est
i
ng
Anal
y
z
et
her
esul

t
soft
het
estt
odet
er
mi
net
hecor
r
ect
nessoft
het
esthy
pot
hesi
s
Cr
eat
eanewt
estbasedont
het
estr
esul
t
.

Ther
esul
t

sf
r
om r
unni
ngmul
t
i
var
i
at
et
est
i
ng
Whi
l
eMVTt
est
i
ngi
spower
f
ul
i
nhel
pi
ngonl
i
nebusi
nessi

ncr
easeconv
er
si
onsr
at
es,
t
her
esul
t
sy
ouwi
l
l
achi
ev
ef
r
om r
unni
ngasi
ngl
et
est
mayv
ar
y
.
Youcanchoosedi

f
f
er
entappr
oachest
odesi
gnandcr
eat
ey
ourmul
t
i
v
ar
i
at
et
est
:

1
2
3

ELEMENTLEVELTESTI
NG:I
nt
hi
st
y

peoft
est
i
ng,
y
out
estdi
f
f
er
entv
ar
i
at
i
onsofanel
ementont
hepage.
Forex
ampl
e,
y
out
est
di
f
f
er
entheadl
i

nev
ar
i
at
i
onsorsev
er
al
i
mages.
Thegoal
ofan“
el
ementl
ev
el
t
est
”i
st
omeasur
et
hei
mpactoft
hatel
ementon
y
ourconv
er
si

onr
at
e.
El
ementl
ev
el
t
est
i
ngi
sconsi
der
edt
heeasi
estt
y
peoft
est
i
ng.
I
tr
equi
r
est
hel
eastamountofef
f
or

t
.
Andi
nmostcases,
el
ementl
ev
el
t
est
i
nghasmi
ni
mum i
mpactony
ourwebsi
t
econv
er
si
onr
at
es.

PAGELEVELTESTI
NG:i
nt
hi
st
y

peoft
est
i
ng,
y
out
estmul
t
i
pl
epageel
ement
satt
hesamet
i
me.
Asanex
ampl
e,
y
oucant
est
di
f
f
er
entpagel
ay
out
s,

and/
oradi
f
f
er
entcombi
nat
i
onofel
ement
sandsoon.
Pagel
ev
el
t
est
i
ngr
equi
r
esmor
eef
f
or
tf
r
om t
he
dev
el

opmentt
eam t
oi
mpl
ementandi
tgener
at
esahi
gheri
mpactony
ourconv
er
si
onr
at
escompar
edt
oel
ementl
ev
el
t
est
i
ng.
Car
ef
ul
l
ydesi

gnedpagel
ev
el
t
est
i
ngcanpr
oduceany
wher
ef
r
om 10%t
o20%i
ncr
easei
nconv
er
si
onr
at
es.

VI
SI
TORFLOW TESTI
NG:i
nt
hi
st
y

peoft
est
i
ng,
y
out
estsev
er
al
nav
i
gat
i
onpat
hsf
orv
i
si
t
or
swi
t
hi
ny
ourwebsi
t
e.
Asan
ex
ampl

e,
anecommer
cewebsi
t
emi
ghtt
estsi
ngl
est
epv
s.
mul
t
i
st
epcheck
out
.
Anot
herex
ampl
ei
st
ot
estdi
f
f
er
entway
s

v
i
si
t
or
scannav
i
gat
ef
r
om cat
egor
ypagest
opr
oductpages.
Vi
si
t
orf
l
owt
est
i
ngcangetcompl
i
cat
edqui
ck
l
y

.
I
tt
y
pi
cal
l
yr
equi
r
esahi
gherl
ev
el
ofef
f
or
tf
r
om y
ourdev
el
opmentt
eam t
o
i
mpl
ement
.
Donecor

r
ect
l
y
,
t
hi
st
y
peoft
est
i
ngwi
l
l
hav
eahi
gheri
mpactony
ourconv
er
si
onr
at
escompar
edt
opagel
ev
el
t

est
i
ng.

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

15


Thedanger
sofmul
t
i
var
i
at
et
est
i
ng
I

fy
ouar
enotcar
ef
ul
wi
t
hpl
anni
ngy
ourmul
t
i
v
ar
i
at
et
est
s,
y
ouwi
l
l
endupwi
t
hweakqual
i
t
yt

est
st
hatt
ak
et
ool
ongt
oi
mpl
ementand
pr
oducenei
t
herr
esul
t
snori
nsi
ght
s.

“

Youmustal
way
sr
emembert
hatt
est
i

ng(
ABormul
t
i
v
ar
i
at
e)i
sonl
yonecomponentofaconv
er
si
onr
at
eopt
i
mi
z
at
i
onwor
k
.
Wehav
eseenmanycompani
est
hatent
i
r

el
yr
el
i
edont
est
i
ngsof
t
war
ewi
t
houtdoi
ngani
ndept
hanal
y
si
sofwhatt
heywer
et
est
i
ng.
Our
2007ar
t
i
cl
eont

hecaseagai
nstmul
t
i
v
ar
i
at
et
est
i
ngpoi
nt
soutt
hi
sex
ampl
e:

Let
’
sdosomesi
mpl
emat
h.

Sayy
ouwantt
ot
estsi

xdi
f
f
er
entel
ement
sonapage(
header
s,
benefit
sl
i
st
,
her
o
shot
s,
cal
lt
oact
i
on,
et
c)
.

Foreachel
ement
,

y
ouwi
l
lchoosef
ourdi
f
f
er
entopt
i
ons.Thi
smeansy
ouwi
l
lhav
ea
t
ot
alof4^
6=4,
096possi
bl
escenar
i
ost
haty
ouwi
l
lhav
et

ot
est
.

“

Asagener
alr
ul
eoft
humb[
bei
ngmor
eaggr
essi
v
e]
,
y
ouwi
l
lneedar
ound200
conv
er
si
onsperscenar
i
ot
oensur

et
hedat
ay
ouar
ecol
l
ect
i
ngi
sst
at
i
st
i
cal
l
y
si
gni
ficant
.Thi
st
r
ansl
at
esi
nt
o4,
096*200=819,
200conv

er
si
ons.

I
fy
ourwebsi
t
econv
er
t
sar
ound1%,
y
ouwi
l
lneed819,
200*100=81,
920,
000vi
si
t
or
s
bef
or
ey
oust
ar
tgai

ni
ngsomeconfidencei
ny
ourt
estr
esul
t
s.

I
ft
est
i
ng4,
096v
ar
i
at
i
onssounddi
f
ficul
t
,
i
magi
nehowcompl
i
cat
edmat

t
er
swi
l
lget
byaddi
ngv
ar
i
at
i
oni
ncampai
gns,
of
f
er
s,
pr
oduct
s,
andkeywor
ds.Yes,
r
unni
ngt
hat
manyt
estv
ar

i
at
i
nsi
snotunhear
doff
ormanyl
ar
gerwebsi
t
es.

Whencr
eat
i
nganMVTt
est
,keept
hesepossi
bl
epr
obl
emsi
nmi
nd:
Beawar
eofcr
eat
i
ngt

het
estwi
t
houtpay
i
ngcl
oseat
t
ent
i
ont
ot
hehy
pot
hesi
sbehi
ndi
t

Bemi
ndf
ul
oft
henumberofv
ar
i
abl
esy
ouar
et

est
i
ngandt
hei
rdependencyononeanot
her
Beawar
eoft
hel
engt
hoft
i
mei
twi
l
l
t
ak
et
ocompl
et
et
het
estt
oast
at
i
st
i
cal

si
gni
ficance

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

16


C H A P TER 3
THEDI
FFERENCEBETWEENA/
BTESTI
NG
ANDMUL
TI
VARI
ATETESTI
NG
WRI
TTENBY:KHALI

DSALEH


C H A P TER 3
THEDI
FFERENCEBETWEENA/
BTESTI
NGANDMUL
TI
VARI
ATETESTI
NG
Bot
hA/
Bt
est
i
ngandmul
t
i
v
ar
i
at
et
est
i
ngshar
et
hesameconcept

:
spl
i
t
t
i
ngv
i
si
t
or
sbet
weensev
er
al
desi
gnsofy
ourwebsi
t
eorwebpaget
o
det
er
mi
newhi
choft
hesedesi
gnsgener
at
esmor

econv
er
si
ons.
So,
whi
choft
heset
womet
hodsi
sbestsui
t
edt
ohel
py
oui
ncr
easey
ourwebsi
t
econv
er
si
onr
at
e?

ABTest
i
ng

ABTest
i
ng(
orSpl
i
tt
est
i
ng)al
l
owsy
out
ot
estoneor
mor
ev
ar
i
at
i
onsf
oreachpageony
ourwebsi
t
e
agai
nsteachot
her
.
Let

’
st
ak
ef
ewex
ampl
es:
Onanecommer
cewebsi
t
e,
y
oucancr
eat
eanew
desi
gnofy
ourpr
oductpaget
ot
estwhet
hert
he
pr
oducti
mageshoul
dbepl
acedont
hel
ef

tort
he
r
i
ghtsi
deoft
hepage.
Onal
eadgener
at
i
onwebsi
t
e,
y
oucanuseA/
Bt
est
i
ngt
ocr
eat
emul
t
i
pl
edesi
gnsofal
andi
ngpagewher

eeachdesi
gnusesdi
f
f
er
ent
headl
i
nesordi
f
f
er
enti
mages.
Onabl
ogsubscr
i
pt
i
onpage,
y
oucant
estdi
f
f
er
entdesi
gnsdi
spl
ay

i
ngassor
t
edcol
or
sandt
ex
tont
hesubscr
i
bebut
t
on.

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

18


Edi
t

ornot
e:
Pl
easenot
et
hatweusebut
t
oncol
or
s,
f
ont
s,
headl
i
nesasawayt
ocl
ar
i
f
yt
heconceptoft
est
i
ng.
Succesf
ul
ABandmul
t
i

v
ar
i
at
et
est
s
shoul
di
ncl
udemor
esophi
st
i
cat
edchangest
oy
ourpage.
Ther
ear
et
wogener
al
appr
oachest
ocr
eat
i
ngABt
est

s:
1.Test
i
ngnew,
r
adi
cal
desi
gnsofanent
i
r
epageorpr
ocess.
Asanex
ampl
e,
t
est
i
ngt
hr
eeent
i
r
el
ydi
f
f
er
entanddi

st
i
nctdesi
gnsofa
homepage
2.Test
i
ngsmal
l
changesonapage:
eachdesi
gnmak
esasl
i
ghtshi
f
ti
nt
hesameel
ement
.
Forex
ampl
e,
eachv
ar
i
at
i
onwi

l
l
t
estdi
f
f
er
ent
her
oi
magesandsoon
I
nourex
per
i
ence,
wer
ecommendusi
ngABt
est
i
ngt
omeasur
et
heper
f
or
manceoft
wot
ofiv

enewdesi
gnscompar
edt
ot
heor
i
gi
nal
desi
gn.
Wet
y
pi
cal
l
ydonotr
unmor
et
hansev
endesi
gnsagai
nstanor
i
gi
nal
page.
I
ti
sal
soi

mpor
t
antt
or
emembert
hatt
hemor
edesi
gns
y
oui
nt
r
oducei
naspl
i
tt
est
:
Thet
estwi
l
l
r
equi
r
emor
et
i
met

or
eachst
at
i
st
i
cal
si
gni
ficance(
concl
ude)
Yourr
esul
t
sar
emor
el
i
k
el
yt
osuf
f
erf
r
om st
at
i
st

i
cal
er
r
or
s
I
nani
deal
wor
l
d,
eachchal
l
engert
ot
hecont
r
ol
wi
l
l
i
ncl
udeonechange.
Howev
er
,
t
hi

sappr
oachdoesnotwor
ki
nr
eal
i
t
y
.
Doi
ngsowi
l
l
r
equi
r
ebot
hal
ongt
i
meandasi
gni
ficantbudget
.
Test
i
ngmor
er
adi
cal

changesi
neachv
ar
i
at
i
oni
sabet
t
erappr
oach.
Att
hesamet
i
me,
i
t
i
si
mpor
t
antt
or
emembert
hat
:
Asi
ngl
eandcoher
enthy

pot
hesi
sdr
i
v
esev
er
ychangey
oumak
eonapage.
Thehy
pot
hesi
scandr
i
v
esev
er
al
changest
osev
er
al
el
ement
sonapage
Radi
cal
changet
oapagedoesNOTmeant

haty
ouar
er
andoml
ychangi
ngel
ement
sont
hepage.
Youshoul
dal
way
st
i
eanychanget
o
y
ourt
esthy
pot
hesi
s
Mul
t
i
v
ar
i
at
et

est
i
ngal
l
owsy
out
ot
estsev
er
al
v
ar
i
at
i
onsof
mul
t
i
pl
eel
ement
sonawebpageal
l
atonce.
Forex
ampl
e,
y
oucant

estmul
t
i
pl
eopt
i
onsoft
hepr
oductname
onanecommer
cepr
oductpage,
al
ongwi
t
hv
ar
i
ous
al
t
er
nat
i
v
esoft
hepr
oducti
mageanddi
f

f
er
entt
y
pesandcol
or
s
ofaddt
ocar
tbut
t
ons.
Youcanal
sousemul
t
i
v
ar
i
at
et
est
i
ngt
oev
al
uat
ev
ar
i

ous
headl
i
nesandsev
er
al
desi
gnsoft
heCTAonal
eadgener
at
i
on
l
andi
ngpage.
MVTt
est
i
ngal
l
owsy
out
oz
oom i
nandf
ocusont
hechanges
y
ouar

emak
i
ngatanel
ementl
ev
el
.

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

19


Whi
l
emul
t
i
v
ar
i

at
et
est
i
ngi
sgr
eati
nt
heor
yt
oanal
y
z
eel
ementl
ev
el
i
mpactonconv
er
si
onr
at
es,
i
thast
hr
eedr
awback
s:

I
tr
equi
r
essi
gni
ficantl
ev
el
soft
r
af
ficorconv
er
si
onst
oconcl
udeat
est
I
fy
ouar
enotcar
ef
ul
whendesi
gni
ngy
ourmul
t

i
v
ar
i
at
et
est
,
y
oucanendupt
est
i
ngt
housandsofdesi
gnsagai
nstt
heor
i
gi
nal
Asi
ngl
emul
t
i
v
ar
i
at
et

esti
ncl
udessev
er
al
si
mul
t
aneouschangest
omul
t
i
pl
eel
ement
s;
t
husi
ti
shar
dt
oi
sol
at
et
heex
actr
easona
par
t

i
cul
ardesi
gnper
f
or
msacer
t
ai
nway
.
I
ti
sal
sodi
f
ficul
tt
oassesst
hei
mpactofel
ement
si
nt
er
act
i
onony
ouror
i

gi
nal
desi
gn
MVTt
est
i
ngi
sv
er
ypower
f
ul
.
Howev
er
,
y
oumustbecar
ef
ul
whenusi
ngi
tt
oopt
i
mi
z
ey
ourconv

er
si
onr
at
es.
Mostcompani
esf
or
getabout
t
hel
ar
gev
ol
umeofv
i
si
t
or
sandconv
er
si
onsr
equi
r
edt
oconcl
udeat
est
.

Asar
esul
t
,
t
heyfindt
hemsel
v
esr
unni
ngt
est
sf
orsev
er
al
week
s
wi
t
houtbr
i
ngi
ngt
hem t
oaconcl
usi
on.
Addi
t

i
onal
l
y
,
si
ncet
est
i
ngsof
t
war
eal
l
owst
hem t
odoso,
manycompani
est
endt
oal
t
er
nat
eel
ement
sr
andoml
ywi
t

houtt
hi
nk
i
ngaboutt
he
r
at
i
onal
ebehi
ndt
hechange.
Thi
smi
st
ak
eal
onei
senought
or
educet
hei
mpact(
orev
enk
i
l
l
)t

hebenefit
sofanyt
est
i
ngpr
ogr
am.

Mul
t
i
var
i
at
et
estvs.anA/
Bt
est
:whatt
ypeoft
estshoul
dyouuse?
Shoul
dy
oust
ar
twi
t
hanA/
Bormul

t
i
v
ar
i
at
et
estf
orapar
t
i
cul
arpage?
Ther
ei
snocor
r
ectanswert
ot
hi
squest
i
on.
A/
Bt
est
sar
eusef
ul
f

ort
est
i
ngal
t
er
nat
edesi
gnsofanent
i
r
epageorapr
ocess.
Weusual
l
y
r
ecommendusi
ngt
hem whi
l
edeci
di
nghi
ghl
ev
el
orr
adi
cal

changest
ot
heopt
i
mi
z
edpage(
orar
ea)
.
MVTt
est
i
ng,
ont
heot
herhand,
al
l
ows
f
orfinegr
ai
nedt
est
i
ngonapar
t
i
cul

arpage.
Theyar
ehel
pf
ul
i
ndet
er
mi
ni
ngt
hemosti
mpact
f
ul
el
ement
sonv
i
si
t
or
s.
Wehi
ghl
yr
ecommendusi
ngABt
est
i

ngi
nt
hef
ol
l
owi
ngscenar
i
os:

Wehi
ghl
yr
ecommendusi
ngABt
est
i
ngi
nt
hef
ol
l
owi
ng
scenar
i
os:

Wehi
ghl

yr
ecommendusi
ngmul
t
i
var
i
at
et
est
i
ngi
nt
he
f
ol
l
owi
ngscenar
i
os:

I
fy
ouar
est
ar
t
i
ngoutwi

t
ht
het
est
i
ngpr
ocess

I
fy
ouhav
ebeenconduct
i
ngABt
est
sf
oratl
east812mont
hs

I
fy
ouhav
el
i
mi
t
ednumberofwebsi
t
ev

i
si
t
or
s

I
fy
ouhav
eal
ar
genumberofmont
hl
yv
i
si
t
or
s

I
fy
ouhav
el
i
mi
t
ednumberofconv
er
si

ons

I
fy
ouhav
eal
ar
genumberofmont
hl
yconv
er
si
ons(
mor
et
han

I
fy
ouar
el
ook
i
ngf
orar
adi
cal
depar
t
ur

ef
r
om y
ourex
i
st
i
ng
desi
gns

10,
000conv
er
si
onspermont
h)
I
fy
ouar
el
ook
i
ngt
ofinet
uneex
i
st
i
ngdesi

gns

Formostcompani
es,
wer
ecommendst
ar
t
i
ngwi
t
hanA/
Bex
per
i
mentt
oassesssi
gni
ficantdesi
gnchangest
ot
hewebsi
t
e.
Howev
er
,
i
fy
oual

r
eadyhav
eanex
i
st
i
ngpage,
y
oucanst
ar
twi
t
hasmal
l
mul
t
i
v
ar
i
at
et
est(
l
esst
han12orsodi
f
f
er
entscenar

i
os)
.
Thegoal
oft
hi
si
ni
t
i
al
t
esti
st
odet
er
mi
newhi
choft
heel
ement
s(
headl
i
ne,
i
mage,
benefitl
i
st

,
et
c.
)r
esonat
esmostwi
t
hv
i
si
t
or
s.
Anal
y
si
soft
hefir
st
t
estr
esul
t
swi
l
l
hel
pgui
det
heneedf

orf
ur
t
herMVTorA/
Bt
est
s.

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

20


Fi
vest
epswhent
est
i
ngyourwebsi
t
e


1
2
3
4
5

Det
er
mi
newhi
chofy
ourwebsi
t
epagesar
egoodcandi
dat
esf
oropt
i
mi
z
at
i
onbyfindi
ngwhi
chof
y
ourpagesar
el

eak
i
ngt
hemostv
i
si
t
or
s.
Cal
cul
at
et
hepot
ent
i
al
r
ev
enuei
mpacty
oucangai
nf
r
om
fix
i
ngt
hel
eak

.

Checkt
henumberofv
i
si
t
or
st
oy
ourwebpage.
Thenumberofv
i
si
t
or
swhowi
l
l
got
hr
ought
heact
ual
webpageorpr
ocessi
mpact
sy
ourt
est

i
ngv
el
oci
t
y
.
Al
t
houghy
oursi
t
emayhav
e500,
000v
i
si
t
or
sa
mont
h,
apar
t
i
cul
arpaget
haty
ouwantt
ot

estmayr
ecei
v
ef
ewerv
i
si
t
or
si
nasi
ngl
emont
h

Donotr
uny
ourt
est
sf
orl
esst
hant
woweek
snorl
ongert
hanf
ourweek
s


Pi
ckt
her
i
ghtconv
er
si
ongoal
.
Notev
er
yt
estshoul
dhav
et
hegoal
ofi
ncr
easi
ngt
hemacr
o
conv
er
si
ons.
Manysuccessf
ul
t
est

shel
pi
ni
ncr
easi
ngmi
cr
oconv
er
si
onr
at
es.
Thi
si
spar
t
i
cul
ar
l
y
i
mpor
t
anti
fy
ourwebsi
t
eorl

andi
ngpagedoesnotr
ecei
v
eenoughconv
er
si
ons.
St
ar
t
i
ngoutwi
t
h
mi
cr
oconv
er
si
ont
est
sal
l
owsy
out
oconductex
per
i
ment

sonasmal
l
erscal
e.

Ex
ami
newhatel
ement
sy
oushoul
dt
est
.
Notal
l
el
ement
sonapagewi
l
l
hav
et
hesamei
mpacton
y
ourconv
er
si
onr

at
e.
Det
er
mi
newhi
chel
ement
swi
l
l
hav
et
hemosti
mpactony
ourbot
t
om l
i
ne
basedont
hev
oi
ceofcust
omerdat
a,
per
sonadev
el
opment

,
andanal
y
t
i
csanal
y
si
s.

THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

21


C H A P TER 4
COMPARI
NGMUL
TI
VARI
ATE&ABTESTI

NGTOOLS
(
UPDATEDFOR2018)
WRI
TTENBY:KHALI
DSALEH


C H A P TER 4
COMPARI
NGMUL
TI
VARI
ATE&ABTESTI
NGTOOLS(
UPDATEDFOR2018)
I
ti
st
i
menowt
osel
ectanABormul
t
i
v
ar
i
at
et

est
i
ngpl
at
f
or
mt
ost
ar
tr
unni
ngy
ourt
est
s.
Ther
ear
eov
ert
hi
r
t
ydi
f
f
er
entt
ool
sav
ai

l
abl
ei
nt
he
mar
k
et
pl
acef
ory
out
ochoosef
r
om.
So,
y
oumi
ghtask
:
Whati
st
her
i
ghtspl
i
tt
est
i
ngt

ool
f
ormywebsi
t
e?
Wi
l
l
Iaf
f
ectt
hesuccessofmyconv
er
si
onopt
i
mi
z
at
i
onpr
ogr
am bychoosi
ngonepar
t
i
cul
art
ool
?

Themar
k
eti
sf
ul
l
oft
est
i
ngt
ool
st
hatr
angef
r
om si
mpl
ef
r
eet
ool
st
osophi
st
i
cat
edent
er
pr
i

sespl
i
tt
est
i
ngsof
t
war
e.
Thi
sar
t
i
cl
ewi
l
l
wal
k
y
out
hr
ought
hepr
osandconsofdi
f
f
er
entspl
i

tt
est
i
ngt
ool
sav
ai
l
abl
e.
Wewi
l
l
al
sosuggestcr
i
t
er
i
at
ohel
py
ousel
ectt
hesof
t
war
et
hatbestfit
sy

ourneeds,
consi
der
i
ngeaseofuse,
t
heagi
l
i
t
yofr
esul
t
s,
costof
r
unni
ng,
t
est
i
ngsuppor
t
,
oper
at
i
onal
i
ssues,

andt
est
i
nganal
y
t
i
cs.

Thedi
f
f
er
entt
ypesofABormul
t
i
var
i
at
et
est
i
ngt
ool
savai
l
abl
e
2018dat

af
r
om dat
any
z
eshowst
hef
ol
l
owi
ngt
opABt
est
i
ngt
ool
s:
Wecl
assi
f
yA/
Bt
est
i
ngt
ool
sbasedondi
f
f
er

entcr
i
t
er
i
a:
Basedont
ool
pr
i
ce
Basedonhowt
hespl
i
tt
est
sar
ei
mpl
ement
edona
websi
t
e
Basedont
hescal
eofsof
t
war
e(

ent
er
pr
i
sev
s.
smal
l
er
scal
esof
t
war
e)
Bef
or
ewedi
scussdi
f
f
er
entway
st
ocl
assi
f
ydi
f
f
er

ent
spl
i
tt
est
i
ngsof
t
war
e,
l
et
’
sst
ar
twi
t
habr
i
efdi
scussi
on
oni
nhouseABt
est
i
ngt
ool
s.


THEDEFI
NI
TI
VEGUI
DETO
MUL
TI
VARI
ATEANDABTESTI
NG

23