A/B Testing Experiments

Measure the impact of Alhena AI by comparing key metrics against alternative support solutions like Zendesk or Intercom.

Overview

The A/B testing experiments feature splits your website visitors into two groups:

• Test Group: Users who see and interact with Alhena AI

• Control Group: Users who don't see Alhena AI (widget is disabled)

How It Works

1. Create an Experiment: Define your audience split (e.g., 50% test, 50% control) in the Analytics dashboard

2. User Assignment: When a visitor loads your website, they are automatically assigned to a group based on their unique fingerprint

3. Consistent Experience: The same user will always be assigned to the same group throughout the experiment

4. Track Metrics: Monitor key metrics like add-to-cart value and checkout revenue for each group

The experiment:loaded Event

When an active experiment is running, the Alhena widget fires the experiment:loaded event after determining which group the user belongs to. This event allows you to:

• Conditionally load other SDKs based on the experiment group

• Customize your page behavior for control group users

• Implement your own tracking logic

Event Signature

window.gleenWidget.on('experiment:loaded', (experimentGroup) => {
    // experimentGroup is either 'test' or 'control'
});

Parameters:

• experimentGroup (string): Either 'test' (user sees Alhena AI) or 'control' (user doesn't see Alhena AI)

Example: Loading Zendesk for Control Group

A common use case is to show an alternative support solution (like Zendesk) to users in the control group who don't see Alhena AI:

<script src="https://app.alhena.ai/js/gleen-widget/main.js"></script>

<script>
    window.gleenWidget.on('experiment:loaded', (experimentGroup) => {
        if (experimentGroup === 'control') {
            // Dynamically load Zendesk for control group users
            const script = document.createElement('script');
            script.id = 'ze-snippet';
            script.src = 'https://static.zdassets.com/ekr/snippet.js?key=7460dcd8-6a51-442e-9675-9781f5954552';
            document.body.appendChild(script);
        }
    });
</script>

This ensures:

• Test group users see Alhena AI

• Control group users see Zendesk instead

• Fair comparison between the two solutions

Important Notes

• The experiment:loaded event only fires when an active experiment is running

• User assignment is deterministic and based on their fingerprint UUID, ensuring consistency across sessions

• Users in the control group will not see the Alhena AI widget (it's automatically disabled)

• Make sure to set up your cart and checkout events before starting an experiment to track revenue impact

Setting Up Experiments

Prerequisites

Before creating an experiment, ensure you have:

1. Cart Events: Implemented using gleenWidget.sendCartEvent()

2. Checkout Events: Implemented using gleenWidget.sendCheckoutEvent()

See the Cart and Checkout Event API documentation for setup instructions.

Creating an Experiment

1. Navigate to Analytics > Experiments in your dashboard

2. Click Create Experiment

3. Enter experiment name and set audience split

4. Verify cart and checkout events are being received

5. Click Start Experiment

Monitoring Results

The experiment dashboard shows real-time metrics for both groups:

• Exposure: Number of unique users in each group

• Add to Cart Value: Total cart value added

• Checkout Revenue: Total revenue from completed checkouts

• Delta: Percentage difference between test and control groups

Ending an Experiment

When you're ready to conclude:

1. Navigate to your experiment in the dashboard

2. Click End Experiment

3. Review the final results and statistical significance

4. The system automatically determines if the experiment was Won, Lost, or Inconclusive

If you don't end the experiment, it will end automatically after 30 days.

How It Works Under the Hood

When a user visits your website:

1. The Alhena widget loads and checks if an experiment is active

2. The user is assigned to a group based on their unique fingerprint using a deterministic algorithm

3. The experiment:loaded event fires with either 'test' or 'control'

4. For control group users, the Alhena widget is automatically disabled

This ensures:

• Consistency: The same user always sees the same experience

• Even distribution: Users are evenly split between groups

• No bias: Assignment is random and not based on user behavior

Best Practices

1. Run for adequate duration: Allow at least 1-2 weeks to gather meaningful data

2. Don't modify mid-experiment: Changing audience split invalidates results

3. Track other metrics: Use the event to send experiment group to your analytics platform

4. Test your integration: Verify the control group experience works as expected

5. Statistical significance: Wait for enough data before drawing conclusions

FAQ

Q: Can I run multiple experiments simultaneously?

A: No, only one experiment can be RUNNING at a time per bot profile.

Q: What happens to users after an experiment ends?

A: All users will see the normal Alhena AI widget (no experiment assignment).

Q: Can I change the audience split after starting?

A: No, this would invalidate the experiment. You'll need to end and create a new one.

Q: Is the event fired on every page load?

A: Yes, the event fires each time the widget loads and detects an active experiment.