How to Measure Whether AI Is Improving the Guest Experience

→ In Part 1 we argued that AI should amplify hotel employees, not replace them.

→ In Part 2 we argued that the more a hotel automates, the more its humanity is worth.

This part is about the question every operator asks next: how do we know it is working?

The scoreboard every hotel knows

Hotels already track many important performance indicators:

Together, revenue, finance and operations teams use these numbers to understand the commercial and operational health of the property, and a hotel that ignores them will not stay in business long enough to worry about anything else.

Guest-satisfaction measures are also common, especially in chain and luxury hotels. Surveys and NPS add an important perspective, but they are usually retrospective and aggregated. They can show that something went well or badly without always revealing which employee action created the outcome, or whether technology helped make that action possible.

And that is the gap. The metrics receiving the most operational attention emphasize revenue, cost, speed and productivity. What few conventional hotel scoreboards show clearly is the thing the first two parts of this series were about: whether technology is giving your people more time and better information to care for guests, and whether that is creating the moments guests remember.

What the stopwatch cannot see

Here is the trap hiding inside a purely efficiency scoreboard. Put a kiosk next to your warmest receptionist and compare them on check-in. In a simple comparison, the kiosk will usually appear faster, cheaper and more consistent. If those are the only outcomes being measured, reducing the receptionist's role can look like the rational decision, and hotels make exactly that decision, believing the numbers told them to.

But efficiency numbers show only one perspective. The same minute means different things to different guests:

And a receptionist with empathy can recognize that nuance and act accordingly. Good hospitality is not uniformly faster or slower. It recognizes what this guest needs in this moment, and gives employees the ability to respond accordingly. A stopwatch cannot see any of that. It can only tell you the minute was spent.

That is the blind spot. When the only instruments you read are efficiency instruments, the human layer of your hotel becomes invisible on paper long before it disappears in person. And as we argued in Part 2, that is the road to the most dangerous quadrant of all: efficient and forgettable.

The second scoreboard

We are not proposing you replace the scoreboard you have. Keep it; it keeps the lights on. We are proposing you hang a second one next to it, with signals that make the human layer visible:

If you are a general manager reading that second column with a skeptical eye, asking "and how exactly would I measure these?", that is the right question, and it deserves an honest answer. Some of this information already exists in your PMS, your CRM, your surveys, your reviews or your service-management tools. In other hotels it is fragmented, inconsistently entered or not recorded at all. The goal is not to build a perfect dashboard immediately. The goal is to choose two or three signals, define them clearly and begin collecting them reliably.

Most hotels already hold some knowledge about their guests. The problem is that it is scattered across the PMS, the CRM, messaging tools, survey comments and, above all, employees' memories. A preference creates value only when it is recorded clearly, available to the right person and used at the right moment.

What matters is not how many notes are written. It is whether appropriate, actionable information reaches the right employee and improves a relevant moment in the guest's stay, with the guest's consent and comfort always ahead of the hotel's curiosity. Measured that way, recognition stops being a vague ideal and becomes a number you can watch grow.

The name test

One simple indicator, we call it the name test, is the share of positive reviews that mention an employee by name. No new software required:

1. 1Read your last hundred reviews.
2. 2Count the positive ones that mention an employee by name.
3. 3Repeat every quarter and watch the direction.

Guests are especially likely to name employees who noticed something, solved a problem, or made them feel recognized rather than processed. Nobody writes "the check-in was 40 seconds faster than expected." Every positive named mention is a ceiling moment that was strong enough for a guest to remember it, spell the name, and publish it to strangers.

Treat it as a signal, not as proof. Review volume, name badges, incentives or a new review platform can all move the number, so look at the direction over time and read the comments behind it. A meaningful decline, especially alongside rapid automation, is not a verdict. It is a reason to investigate whether personal service is becoming less visible in your hotel.

Where the freed hour goes

The other question we would keep asking is a management question rather than a dashboard number: when automation takes translation, routing, lookup and chasing off a shift, where does the recovered time go? Automation rarely hands you a neat, labeled hour. It saves two minutes here and five minutes there, across dozens of small tasks, and if nobody pays attention those minutes quietly leak back into busywork.

There is a well-known lesson from outside our industry here. Google's famous "20 percent time" let engineers spend part of their week on business-related ideas beyond their core assignments, and Google has credited it with projects including Gmail and Google News. The point was never the free time itself. The point was that discovery needs room, and a calendar packed edge to edge produces no discoveries.

Hotels can apply the same principle on a smaller, more practical scale. An employee who has time to genuinely listen to a guest, not merely complete a transaction, notices things. The breakfast dish guests quietly skip. The airport-transfer worry that keeps coming up at checkout. The small service nobody asked for out loud but everyone wants. Those observations are a source of insight no competitor can access in exactly the same way: your people, learning directly from your guests.

But freed time does not become innovation automatically. It needs a simple habit:

• A short weekly frontline conversation
• A place where recurring observations are written down
• Permission to test small improvements

Give the freed hour that structure, and it may become one of the hotel's most valuable sources of continuous improvement.

Measure the conditions, not the warmth

One warning before you build the second scoreboard. The moment you turn humanity itself into a quota, you kill it. A scripted "sincere" greeting measured by a checklist is worse than no greeting at all. Staff feel the difference immediately, and guests feel it a moment later.

So do not measure the smile. Measure the conditions that make the smile possible, and those conditions do have honest numbers:

• Staff retention
• Employee engagement scores
• Percentage of service issues staff resolve without needing a manager's approval
• Average time from complaint to recovery
• Frontline observations submitted and acted upon

A team that stays, that is trusted to act, and that sees its ideas taken seriously is far more likely to deliver genuine warmth. The warmth itself must remain unmeasured and unforced. That is precisely what makes it worth something.

Start with three signals

A hotel does not need to measure everything at once. Start with three:

1. 1the percentage of positive reviews naming an employee,
2. 2the percentage of recorded preferences successfully used,
3. 3the percentage of complaints genuinely resolved before checkout.

Review them alongside ADR, RevPAR and labor cost, not instead of them. That is enough to begin seeing whether technology is merely making the hotel more efficient, or genuinely helping your people make the stay more valuable.

The scoreboard decides the hotel

Technology raises the floor. Only people raise the ceiling. Your metrics are the lever between the two, because your team will optimize whatever you put on the wall. Put only the stopwatch on the wall and you will get a faster, cheaper, emptier hotel. Put both scoreboards up and you will get what this framework has been arguing for from the beginning: machines doing what was never the point, and people, visibly and measurably, doing what always was.

Choose your scoreboard carefully. It is quietly choosing your hotel.