A/B Testing and Experiments Issues

This guide helps you troubleshoot problems with ShahiLandin’s A/B testing and experiment features.

Cannot Create Experiment

Symptom

Unable to create new A/B test or receive errors when attempting to set up experiments.

Solutions

Problem 1: Experiments Feature Not Enabled

Solution:

1. Go to ShahiLandin → Settings → Features
2. Find A/B Testing Experiments
3. Check Enable Experiments
4. Click Save Changes
5. Page will reload with experiments functionality available

Problem 2: Insufficient Permissions


Symptom: “You do not have permission to create experiments” error.

Solution:

6. Only administrators and editors can create experiments by default
7. Check your user role in Users → Your Profile
8. If you need to grant permissions to other roles:

– Add to theme’s functions.php:
`php
addfilter(‘shahilandinexperiment_capability’, function($cap) {
return ‘edit_posts’; // Or any capability
});
`

Problem 3: Parent Landing Page Not Published


Symptom: Can’t select landing page for experiment.

Solution:

9. Experiments require a published parent landing page
10. Go to Landing Pages
11. Ensure the landing page is published (not draft)
12. Try creating experiment again

Problem 4: Database Tables Missing


Symptom: General error when creating experiment.

Solution:

13. Experiment data stored in wpshahilandinexperiments table
14. Check if table exists in phpMyAdmin
15. If missing, run migrations:

– Deactivate and reactivate plugin
– Or use WP-CLI: wp shahilandin migrate

16. Contact support if issue persists

Variants Not Displaying


Symptom


Created experiment with variants but visitors only see original version.

Solutions


Problem 1: Experiment Not Started


Solution:

17. Go to Landing Pages → Experiments
18. Find your experiment
19. Check Status column
20. If “Draft” or “Scheduled”, click Start Experiment
21. Confirm starting the test

Problem 2: Traffic Split Not Configured


Symptom: All traffic goes to control (original).

Solution:

22. Edit experiment
23. Check Traffic Allocation section
24. Ensure traffic split is configured:

– Control: 50%
– Variant A: 50%
– (Or any distribution totaling 100%)

25. Update experiment
26. Test in incognito window

Problem 3: Variant Assignment Logic Issue


Solution:

27. Variants assigned based on cookie/session
28. If cache is too aggressive, same version always shown
29. Exclude experiments from page cache:

– Settings → Performance
– Uncheck Cache Experiment Pages
– Or add to caching plugin exclusions

30. Clear all caches:

– Browser cache
– Plugin cache
– CDN cache

Problem 4: Testing While Logged In


Symptom: Always see same variant when logged in.

Solution:
By default, logged-in admins are “sticky” to prevent skewing results:

31. You’ll see the same variant consistently
32. To test different variants:

– Test in different browsers
– Use incognito/private windows
– Log out and clear cookies

33. Or disable sticky sessions temporarily:

`php
addfilter(‘shahilandinexperimentstickyassignment’, ‘_returnfalse’);
`

Statistics Not Recording


Symptom


Experiment running but no statistics being collected.

Solutions


Problem 1: Analytics Disabled


Solution:
Experiments depend on analytics tracking:

34. Go to Settings → Analytics
35. Ensure Enable Analytics Tracking is checked
36. Save Changes
37. Statistics should start recording within minutes

Problem 2: Insufficient Traffic


Symptom: Experiment shows zeros or very low numbers.

Solution:

38. Verify landing page is receiving traffic:

– Check landing page statistics separately
– If no traffic, experiment can’t collect data

39. Drive traffic to landing page:

– Share on social media
– Run ads
– Send email campaign

40. Wait for statistical significance (see below)

Problem 3: Cache Preventing Tracking


Symptom: Views tracked on regular pages but not experiment variants.

Solution:

41. Disable page caching for experiments:

– Caching plugins often cache experiment pages
– Add experiment URLs to exclusions
– Example URL pattern: /landing-page/?variant=*

42. For W3 Total Cache:

– Performance → Page Cache → Advanced
– Never cache the following pages:
`
/landing-page/
`

43. For WP Rocket:

– Settings → Advanced Rules
– Never Cache URLs:
`
/landing-page/(.*)
`

Problem 4: Cookie Consent Blocking


Symptom: Users who decline cookies not tracked.

Solution:

44. Experiment tracking uses cookies for variant assignment
45. Respect user consent:

– Users who decline may not be included in experiment

46. Alternative: Use server-side assignment (advanced, requires customization)

Winner Declaration Issues


Problem 1: Cannot Declare Winner


Symptom: “Declare Winner” button disabled or not available.

Solution:

47. Insufficient statistical significance:

– Need minimum sample size and confidence level
– ShahiLandin requires:
– At least 100 views per variant
– 95% confidence level
– Detectable difference between variants

48. Check experiment dashboard:

– View experiment details
– Look for “Statistical Significance” indicator
– If not reached, continue running experiment

49. Manual winner declaration (override):

`bash
wp shahilandin experiment declare-winner
`

Problem 2: Winner Declared But Nothing Changes


Symptom: Declared winner but original version still shows.

Solution:

50. After declaring winner, implement the winning variant:

– Edit parent landing page
– Replace content with winning variant content
– Publish changes

51. ShahiLandin doesn’t auto-replace content (by design)
52. You control when/how to implement winning changes

Conversion Tracking Not Working


Symptom


Views tracked but conversions show zero for all variants.

Solutions


Problem 1: Conversion Goals Not Defined


Solution:

53. Edit experiment
54. Set Conversion Goal:

– Goal Name: e.g., “Form Submit”
– Goal Type: Click, Form Submit, or Custom Event

55. Update experiment
56. Ensure landing page has conversion tracking:

`html






Buy Now
`

Problem 2: Conversion Tracking Code Missing


Solution:
Add conversion tracking manually to landing page:
`html

`

Problem 3: Conversion Goal Mismatch


Symptom: Tracking conversions but not showing in experiment stats.

Solution:
Ensure goal names match exactly:

57. Experiment goal name: form_submit
58. Tracking code goal name: form_submit
59. Case-sensitive, must match exactly

Variant Content Issues


Problem 1: Variants Look Identical


Symptom: Created different variants but they look the same on frontend.

Solution:

60. Verify variant differences:

– Edit experiment
– Check each variant’s content
– Ensure changes are substantial enough to notice

61. Clear cache:

– Browser cache
– Plugin cache
– CDN cache

62. Test in incognito:

– Open multiple incognito windows
– Each should show different variant
– Look for the differences you configured

Problem 2: Cannot Edit Variant Content


Symptom: Variant editor doesn’t save changes.

Solution:

63. Check browser console (F12) for JavaScript errors
64. Disable other plugins temporarily (conflict test)
65. Ensure sufficient permissions (admin/editor role)
66. Try different browser
67. If using custom variant storage, check database:

`sql
SELECT * FROM wpshahilandinexperiments WHERE id = ;
`

Problem 3: Variant HTML/CSS Broken


Symptom: Variant displays but layout is broken.

Solution:

68. Each variant inherits parent page structure
69. Make sure variant changes are compatible:

– CSS changes should be scoped
– HTML changes should maintain structure

70. Test variant individually:

– Add ?variant=variant-a to URL
– Check if variant displays correctly
– Fix HTML/CSS issues in variant editor

Traffic Distribution Problems


Problem 1: Uneven Traffic Split


Symptom: 50/50 split configured but one variant gets 70% of traffic.

Solution:

71. Statistical variance is normal:

– Small sample sizes show uneven distribution
– Distribution evens out over time
– Need at least 1000+ total visitors

72. Check assignment logic:

– Use WP-CLI to verify:
`bash
wp shahilandin experiment stats
`

73. Cache issues:

– If page cached, same variant served repeatedly
– Disable caching for experiment pages (see above)

Problem 2: Everyone Gets Same Variant


Symptom: No distribution happening at all.

Solution:

74. Cookie persistence:

– Users assigned to variant via cookie
– Same user always sees same variant (by design)
– Test with different browsers/devices

75. Assignment logic disabled:

– Check experiment status (must be “Running”)
– Verify traffic split adds up to 100%

76. Debug assignment:

`javascript
// Add to landing page temporarily
console.log(‘Assigned Variant:’, shahilandinExperiment.getCurrentVariant());
`

Duration and Timing Issues


Problem 1: Experiment Runs Too Long


Symptom: Experiment running for weeks with no clear winner.

Solution:

77. Need more traffic:

– Calculate required sample size
– Use calculator: https://www.optimizely.com/sample-size-calculator/
– If traffic is too low, either:
– Run longer
– Drive more traffic
– Declare manual winner

78. Differences too small:

– If variants are very similar, harder to detect winner
– Consider more dramatic changes

79. Set maximum duration:

`bash

Stop experiment after 30 days regardless


wp shahilandin experiment schedule-stop –days=30
`

Problem 2: Experiment Stops Automatically


Symptom: Experiment stops before reaching significance.

Solution:

80. Check if maximum duration was set
81. Check if error occurred:

– View experiment in admin
– Look for error messages

82. Restart experiment:

`bash
wp shahilandin experiment start
`

Data and Reporting Issues


Problem 1: Confidence Level Not Showing


Symptom: Statistics show but no confidence indicator.

Solution:

83. Insufficient data:

– Need minimum 100 views per variant
– Need at least some conversions

84. Wait for more data or drive additional traffic

Problem 2: Conflicting Results


Symptom: ShahiLandin shows different winner than external tools.

Solution:
Different tools may use different statistical methods:

• ShahiLandin: Chi-square test, 95% confidence
• Google Optimize: Bayesian inference
• VWO: Frequentist approach

Results may vary slightly. Trust the tool you’re primarily using.

Problem 3: Cannot Export Data


Symptom: Want to export experiment data for external analysis.

Solution:
Use WP-CLI to export:
`bash

Export experiment data to CSV


wp shahilandin experiment export –format=csv > experiment-data.csv

Get detailed statistics


wp shahilandin experiment stats –format=json
`

Multivariate Testing Issues


Problem: Testing Multiple Changes Simultaneously


Symptom: Want to test headline AND button color but ShahiLandin only supports A/B testing.

Solution:

• ShahiLandin focuses on simple A/B testing
• For multivariate testing:

– Run sequential tests:
– Test headline first (A vs B)
– Implement winner
– Then test button color (A vs B)

– Create combined variants:
– Variant A: New headline + Blue button
– Variant B: New headline + Red button
– Variant C: Old headline + Blue button
– (Creates full factorial design)

Integration with Other Tools


Problem 1: Google Optimize Conflict


Symptom: Running both ShahiLandin experiments and Google Optimize causes issues.

Solution:

• Don’t run experiments on same page:

– Use ShahiLandin OR Google Optimize per page
– Not both simultaneously

• If both needed:

– Disable ShahiLandin experiments for specific pages
– Add to page meta or use filter:
`php
addfilter(‘shahilandindisableexperiments’, function($disabled, $postid) {
if ($post_id == 123) { // Page with Google Optimize
return true;
}
return $disabled;
}, 10, 2);
`

Problem 2: Analytics Discrepancy


Symptom: ShahiLandin shows different numbers than Google Analytics.

Solution:

• Different tracking methods:

– ShahiLandin: First-party tracking
– GA: May be blocked by ad blockers

• Check tracking implementation:

– Ensure GA tracking also sends variant data
– Add custom dimension in GA for variant

• Accept small differences as normal

Advanced Troubleshooting


Enable Experiment Debug Mode


Add to landing page HTML (testing only):
`html

`

This logs all experiment decisions to browser console.

Check Experiment Data in Database


`sql
— Get experiment details
SELECT * FROM wpshahilandinexperiments WHERE id = ;

— Get variant assignments
SELECT variant_id, COUNT(*) as assignments
FROM wpshahilandinanalytics
WHERE postid = AND experimentid =
GROUP BY variant_id;

— Get conversions by variant
SELECT variant_id, COUNT(*) as conversions
FROM wpshahilandinanalytics
WHERE post_id =
AND experiment_id =
AND goal IS NOT NULL
GROUP BY variant_id;
`

Use WP-CLI for Experiment Management


`bash

List all experiments


wp shahilandin experiment list

Get experiment details


wp shahilandin experiment get

Start experiment


wp shahilandin experiment start

Stop experiment


wp shahilandin experiment stop

Delete experiment


wp shahilandin experiment delete

Get statistics


wp shahilandin experiment stats

Declare winner


wp shahilandin experiment declare-winner

Delete all variants


wp shahilandin experiment delete-variants
`

Best Practices for Reliable Experiments


To avoid common issues:

• ✅ Set clear hypothesis before starting experiment
• ✅ Calculate required sample size before starting
• ✅ Disable caching for experiment pages
• ✅ Test one change at a time for clear results
• ✅ Run until statistical significance reached
• ✅ Monitor regularly for issues
• ✅ Document results for future reference
• ✅ Implement winner promptly after declaring
• ✅ Archive completed experiments to keep data clean
• ✅ Don’t stop early even if one variant seems to win

When to Stop an Experiment


Stop an experiment when:

• ✅ Statistical significance reached (95% confidence)
• ✅ Minimum sample size achieved (100+ per variant)
• ✅ Clear winner emerged
• ✅ Maximum duration reached (e.g., 30 days)
• ✅ Business needs require decision

Don’t stop when:

• ❌ One variant ahead after first few visitors
• ❌ Daily traffic too low (wait for more data)
• ❌ Results fluctuating day-to-day

Getting Help


If experiment issues persist:

• Check experiment status in admin dashboard
• Review experiment logs (if debug enabled)
• Test in isolation (disable other plugins)
• Verify analytics working on regular landing pages first
• Export data and share with support
• Provide details:

– Experiment ID
– Expected behavior vs actual
– Sample size and duration
– Screenshots of configuration

—

Related Articles:

• A/B Testing Experiments Feature Guide
• Analytics Tracking Issues
• Cache and Plugin Conflicts