A/B Testing & Experiments

ShahiLandin includes a powerful A/B testing system that allows you to create experiments, test variations of your landing pages, and identify the best-performing version based on real data.

What is A/B Testing?

A/B testing (also called split testing) is a method of comparing two versions of a landing page to determine which one performs better. Visitors are randomly assigned to see either version A or version B, and their behavior is tracked to calculate which variant has a higher conversion rate.

Benefits:

1. Data-driven decision making
2. Improved conversion rates
3. Better understanding of your audience
4. Continuous optimization

Enabling A/B Testing


Global Settings


5. Navigate to Settings > ShahiLandin
6. Find the Experiments section
7. Toggle Experiments Enabled to ON
8. Click Save Changes

Default State: A/B testing is disabled by default to prevent accidental experiments.

Requirements


To use A/B testing, you need:

9. Analytics enabled (to track conversions)
10. At least 2 published landing pages
11. The manageshahilandings capability

Creating an Experiment


Method 1: Via Dashboard


12. Go to Landing Pages > All Landing Pages
13. Find the landing page you want to test (Control/Original)
14. Hover and click Create Experiment
15. Configure experiment settings
16. Create the variant page
17. Launch the experiment

Method 2: Via Meta Box


18. Edit a published landing page
19. Find the ShahiLandin Experiments meta box
20. Click Create New Experiment
21. Enter experiment details:

– Experiment Name: e.g., “Headline Test – Nov 2025”
– Variant: Choose an existing page or create a new one
– Traffic Split: Set percentage for control/variant (e.g., 50/50)
– Duration: Number of days to run the test

22. Click Start Experiment

Experiment Settings


Control Page:

23. The original landing page (baseline)
24. Receives a portion of traffic based on split percentage

Variant Page:

25. The alternative version you’re testing
26. Should differ in only ONE element for accurate results

Traffic Split:

27. 50/50: Equal traffic to both versions (most common)
28. 75/25: More traffic to control, safer for high-value pages
29. 90/10: Minimal risk testing, useful for radical changes

Duration:

30. Recommended: 14-30 days for statistical significance
31. Minimum: 7 days
32. Consider your traffic volume when setting duration

Goal Tracking:

33. Form submissions
34. Button clicks
35. Custom conversion events

How Experiments Work


Visitor Assignment


When a visitor lands on a page with an active experiment:

36. The plugin checks if they’ve been assigned before (via cookie)
37. If new visitor, randomly assign to control or variant based on traffic split
38. Store assignment in cookie (shahilandinvariant{post_id})
39. Redirect to assigned variant (if not control)
40. Cookie lasts 30 days to ensure consistent experience

Example Flow:
`
Visitor arrives at /landing/signup
→ No assignment cookie found
→ Random selection: Variant (50% probability)
→ Cookie set: shahilandinvariant123 = 456
→ Redirect to /landing/signup-variant
→ Track view and conversions on variant page
`

Consistent User Experience


Once assigned:

41. Visitor always sees the same version
42. Cookie persists for 30 days
43. Even if they leave and return, they see the same variant
44. Ensures accurate conversion tracking

Data Collection


For each variant, the plugin tracks:

45. Views: Total visitors assigned to this variant
46. Conversions: Goal completions (form submissions, etc.)
47. Conversion Rate: Conversions / Views × 100
48. Timestamp: When each event occurred

Managing Active Experiments


Viewing Experiment Status


Check active experiments:

49. Go to Landing Pages > Experiments
50. See all running, completed, and paused experiments

Experiment List Shows:

51. Experiment name
52. Control and variant pages
53. Traffic split
54. Current performance (views, conversions, conversion rate)
55. Days remaining
56. Status (Running, Paused, Completed)

Monitoring Performance


View real-time experiment results:

57. Edit the control or variant page
58. Find the ShahiLandin Experiments meta box
59. Review current statistics:

– Control: Views, conversions, conversion rate
– Variant: Views, conversions, conversion rate
– Leader: Which version is currently winning

Statistical Significance:

60. The plugin calculates confidence level
61. Displays “Statistically Significant” badge when confidence > 95%
62. Wait for significance before declaring a winner

Pausing an Experiment


Temporarily stop an experiment:

63. Go to Landing Pages > Experiments
64. Find the experiment
65. Click Pause
66. All traffic now goes to control page

When to Pause:

67. Unexpected issues with variant
68. Need to make changes to pages
69. Seasonal break in traffic

Resume:

70. Click Resume to continue the experiment
71. Assignments and data are preserved

Ending an Experiment


Complete and analyze an experiment:

72. Go to Landing Pages > Experiments
73. Find the completed or running experiment
74. Click End Experiment
75. Review final results
76. Choose the winner:

– Keep Variant: Replace control with winning variant
– Keep Control: Discard variant, continue with original
– Keep Both: Maintain both pages separately

Analyzing Experiment Results


Key Metrics


Conversion Rate:

77. Most important metric for winners
78. Formula: (Conversions / Views) × 100
79. Example: 50 conversions from 1000 views = 5% conversion rate

Confidence Level:

80. Statistical measure of result reliability
81. 95%+ confidence = winner is likely not due to chance
82. Below 95% = need more data

Sample Size:

83. Minimum 100 conversions per variant recommended
84. Larger samples = more reliable results
85. Low traffic sites need longer test durations

Winner Declaration


The plugin automatically recommends a winner when:

86. Minimum sample size reached (100 conversions per variant)
87. Confidence level exceeds 95%
88. Conversion rate difference is substantial (>5% relative improvement)

Example Winning Criteria:
`
Control: 1000 views, 40 conversions (4.0%)
Variant: 1000 views, 50 conversions (5.0%)
Relative Improvement: 25%
Confidence: 97%
Recommendation: Variant is the winner
`

Exporting Results


Export experiment data for reporting:

89. Go to Landing Pages > Experiments
90. Click Export next to an experiment
91. Download CSV with detailed data:

– Daily breakdown of views and conversions
– Hourly trends
– Visitor browser, device, referrer data

Best Practices for A/B Testing


Test One Element at a Time


Good Testing:

92. Change headline only
93. Change CTA button color only
94. Change hero image only

Bad Testing:

95. Change headline + button + images + layout
96. Too many variables make results meaningless

Set a Clear Hypothesis


Before starting, define:

Hypothesis Template:
“Changing [ELEMENT] from [CONTROL] to [VARIANT] will improve [GOAL] because [REASON]”

Example:
“Changing the CTA text from ‘Submit’ to ‘Get Free Trial’ will improve form submissions because it’s more specific and value-focused”

Allow Sufficient Time


Minimum Duration:

97. At least 1 business cycle (7 days for B2B, may vary)
98. Run through weekdays and weekends
99. Avoid holidays and special events

Sufficient Traffic:

100. Need at least 100 conversions per variant
101. With 2% conversion rate, need 5000 visitors per variant
102. Calculate required duration: 10,000 visits / daily traffic

Avoid Testing During Outliers


Pause experiments during:

103. Major sales or promotions
104. Site-wide technical issues
105. Unusual traffic spikes (viral content, press)
106. Seasonal events that skew behavior

Document Everything


Keep records of:

107. Hypothesis for each test
108. Exact changes made to variant
109. Start and end dates
110. Final results and winner
111. Learnings and next steps

Advanced Experiment Features


Multi-Variant Testing (A/B/C/D)


Test more than 2 versions:

112. Create primary experiment (A vs B)
113. Add additional variants:

– Click Add Variant in Experiments meta box
– Create variant C, D, etc.

114. Set traffic split for all variants (e.g., 25/25/25/25)

Note: Multi-variant tests require more traffic and longer duration.

Segment-Based Experiments


Target experiments to specific audiences:

Geographic Targeting:
`php
// Show variant only to US visitors
addfilter(‘shahilandinexperimentgeotarget’, function($countries, $experiment_id) {
return [‘US’]; // Only show to US traffic
}, 10, 2);
`

Device Targeting:
`php
// Show variant only to mobile users
addfilter(‘shahilandinexperimentdevicetarget’, function($devices, $experiment_id) {
return [‘mobile’]; // Only mobile gets variant
}, 10, 2);
`

Scheduled Experiments


Launch experiments at a future date:

115. Create experiment as usual
116. Set Start Date to future date/time
117. Experiment automatically begins at scheduled time

Use Cases:

118. Coordinate with marketing campaigns
119. Test during specific seasons
120. Automate weekend vs weekday variants

Automatic Winner Selection


Enable automatic winner implementation:

121. Go to Settings > ShahiLandin > Experiments
122. Enable Auto-Select Winners
123. Set confidence threshold (default: 95%)
124. When threshold reached, variant automatically replaces control

Safety Features:

125. Notification email sent before auto-selection
126. 24-hour grace period to review
127. Option to override automated decision

WP-CLI Commands for Experiments


List Active Experiments


`bash
wp shahilandin experiments list
`

Shows all active experiments with status.

Start an Experiment


`bash
wp shahilandin experiments start –control=123 –variant=456 –split=50/50 –duration=14
`

End an Experiment


`bash
wp shahilandin experiments end –id=789 –winner=variant
`

Export Experiment Data


`bash
wp shahilandin experiments export –id=789 –format=csv > results.csv
`

Experiment Settings Reference


| Setting | Default | Description |
|———|———|————-|
| Experiments Enabled | false | Master switch for A/B testing |
| Default Traffic Split | 50/50 | Default control/variant percentage |
| Default Duration | 14 days | How long experiments run |
| Auto-Select Winners | false | Automatically implement winning variant |
| Confidence Threshold | 95% | Required confidence for winner declaration |
| Minimum Sample Size | 100 | Minimum conversions before declaring winner |

Troubleshooting Experiments


Visitors See Inconsistent Versions


Issue: Same visitor sees different variants on different visits

Solution:

128. Clear cookies and test again
129. Check cookie settings in browser
130. Verify cookie domain configuration

Low Sample Size


Issue: Not enough data to reach significance

Solution:

131. Extend experiment duration
132. Increase traffic to landing page
133. Lower confidence threshold (not recommended)
134. Consider running longer tests

Variant Not Loading


Issue: All traffic goes to control, variant never loads

Solution:

135. Verify experiment is active (not paused)
136. Check variant page is published
137. Review traffic split settings
138. Test in incognito mode

Statistical Significance Not Reached


Issue: Test runs for weeks without clear winner

Solution:

139. Check conversion rate difference (may be too small)
140. Increase sample size by extending duration
141. Consider testing a more dramatic change
142. Review if test is powered correctly for your traffic

Tips for Successful A/B Testing


143. Start with high-impact elements: Test headlines and CTAs first
144. Wait for significance: Don’t end tests early based on hunches
145. Document learnings: Build a knowledge base of what works
146. Test continuously: Always have an experiment running
147. Respect user assignment: Don’t override cookie-based assignments
148. Monitor for issues: Check both variants regularly for errors
149. Celebrate wins: Implement winning variants quickly

—

For analytics integration with experiments, see the Analytics & Tracking article.