--- title: The September 2026 readability study desc: Twenty five hospitality homepages asked for their front page by an honestly identified HTTP client. Eight answered. Method, per-site results and limitations. order: 5 --- # How readable is hospitality, actually **Answer: eight of twenty five sites returned readable content to a plainly identified HTTP client. The other seventeen returned an error, a timeout, or a page with no text in it. Among the eight that answered, the median HTML weight was 641 KB, the median text payload was 7.7 KB, and the median ratio between them was 78 to 1.** Those three medians come from three separately sorted columns. No single page in the sample holds all three values, and quoting them as one page would be wrong. Measured 4 September 2026. Raw results: [probe-2026-09-04.json](/research/probe-2026-09-04.json). ## Method A single GET to each origin's homepage from one datacenter IP, no JavaScript execution, no cookies, no retries, 25 second timeout. Two user agent strings were tried against every site: - **browser**, a current Chrome string - **agent**, `TxtFirstProbe/0.1 (+https://txtfirst.com/probe; research on agent readability)`, honest and contactable. That URL resolves: see [probe](/probe) for what it requests and how to block it. Text was extracted by removing `script`, `style`, `noscript`, `svg`, `template` and `iframe` elements, then all remaining tags, then collapsing whitespace. A result counts as **no content** when the request failed or when fewer than 1,500 bytes of text survived extraction. The threshold is arbitrary and is stated so you can disagree with it. ## Results, agent user agent | site | result | status | html KB | text KB | ratio | |---|---|---|---|---|---| | accor.com | readable | 200 | 728.1 | 5.5 | 131.8x | | airbnb.com | readable | 200 | 1,152.0 | 8.4 | 137.1x | | barcelo.com | readable | 200 | 553.7 | 20.2 | 27.5x | | kayak.com | readable | 200 | 1,539.9 | 18.2 | 84.5x | | mandarinoriental.com | readable | 200 | 1,353.6 | 6.9 | 197.3x | | shangri-la.com | readable | 200 | 112.4 | 3.1 | 36.1x | | tripadvisor.com | readable | 200 | 416.7 | 5.8 | 71.6x | | wyndhamhotels.com | readable | 200 | 292.6 | 13.6 | 21.5x | | agoda.com | no content | 200 | 192.5 | 0.0 | - | | booking.com | no content | 202 | 3.9 | 0.0 | - | | choicehotels.com | no content | TimeoutError | - | - | - | | expedia.com | no content | 429 | - | - | - | | fourseasons.com | no content | 403 | - | - | - | | hilton.com | no content | 403 | - | - | - | | hotels.com | no content | TimeoutError | - | - | - | | hyatt.com | no content | 403 | - | - | - | | iberostar.com | no content | 403 | - | - | - | | ihg.com | no content | 403 | - | - | - | | marriott.com | no content | 403 | - | - | - | | melia.com | no content | 403 | - | - | - | | nh-hotels.com | no content | 403 | - | - | - | | radissonhotels.com | no content | 403 | - | - | - | | riu.com | no content | 200 | 5,821.9 | 0.8 | - | | trivago.com | no content | 403 | - | - | - | | vrbo.com | no content | 429 | - | - | - | ## What stands out **Identifying yourself honestly changed almost nothing, and what it changed looks like noise.** The agent string got through on 8 of 25, the Chrome string on 7. Those totals hide the more interesting detail: three sites flipped. The agent string won airbnb.com and tripadvisor.com and lost trivago.com, which answered Chrome with a 200 and the probe with a 403. With one request per cell and no repeats, a difference of one site is not a finding. What can be said is narrower and still worth saying: there is currently no mechanism by which a well behaved reader can be treated better than a badly behaved one, and nothing in this sample suggests one exists. **A success code is not an answer.** agoda.com and riu.com returned 200 and booking.com returned 202, and none of the three carried readable text. A status code does not measure whether you were understood. riu.com sent 5,822 KB to deliver 0.8 KB of text. **The ratio is worse than the raw weight suggests.** The leanest site in the sample, wyndhamhotels.com, still spends 21.5 bytes for every byte of text. The heaviest that still answers, mandarinoriental.com, spends 197.3. ## The 403 on our own robots.txt From the moment its Cloudflare zone was created until it was found and fixed on 4 September 2026, txtfirst.com returned **HTTP 403 to a client identifying as `Python-urllib/3.14`**, on `/`, on `/spec.md` and on `/robots.txt`. That last one is the part worth keeping: **the site denied a machine the one file that exists to tell machines what they are allowed to do.** Nobody configured that. It is Cloudflare's Browser Integrity Check, which is on by default on a new zone. It was never switched on, it was never off. Measured on the same day across thirteen unrelated domains on separate zones created at different times and untouched for this, the block is narrow: `Python-urllib` gets 403 while `curl`, `python-requests`, `Googlebot`, `GPTBot` and `ClaudeBot` all get 200. So the honest version of this is not that the web refuses machines. It is smaller and, for anyone writing a client, worse: **A site can refuse the Python standard library's default HTTP client without anyone deciding to, and you cannot see it from a browser.** `urllib` is what `urllib.robotparser` uses to fetch `robots.txt`, so a well behaved script that checks permission before crawling is the one that gets turned away, while the same script with a made up user agent string sails through. Politeness was the failure mode. One consequence for the table above: some share of the seventeen non-answers may be an inherited default rather than a decision. This study cannot tell those apart, and the sentence in the limitations below that attributes them to datacenter IP mitigation is one guess among at least two. Separating them takes the same sweep run with a curl user agent from a residential address, which has not been done. And whatever you believe your machine readability policy is, it is worth checking with the client you would actually write: ``` curl -A "Python-urllib/3.14" -o /dev/null -w "%{http_code}" https://yoursite/robots.txt ``` Browser Integrity Check has been turned off here, and the conformance script now checks three plain user agent strings on every run. ## Limitations, stated plainly **The sample is small and every cell is a single observation.** Twenty five sites, one request per site per user agent, no repeats. Eight readable results is not enough to support a claim about a difference of one or two sites, which is why the section above walks its own conclusion back. The medians are medians of eight numbers and should be read as an order of magnitude, not a benchmark. It is also one IP without a browser engine. A real agent driving a headless browser from a residential address would see more than this probe did, and some of the 403s are plausibly bot mitigation reacting to a datacenter IP, or an inherited default like the one described above, rather than a deliberate policy about agents. This probe measured none of those causes and the adverbs in earlier drafts were doing work the data did not support. **The honest reading is that this measures how the web answers a cheap client, not how it answers every agent.** What would replace these figures: the same probe run monthly over a larger sample, from three IP classes, with and without a browser engine, and with enough repeats per cell to separate policy from noise. Until that exists, treat the numbers above as a first look and not as an index. ## Reproduce it The probe is a single Python file with no dependencies. Run it against your own sites. It writes a dated JSON file and prints the medians, the sites that flip between user agents, and a reminder that n is 1 per cell. ``` cd research && python3 probe.py ``` It writes `probe-.json` into the directory you run it from and prints the medians, the sites that answer differently to each user agent, and a reminder that n is 1 per cell. Source: [probe.py](/research/probe.py). The run behind this page: [probe-2026-09-04.json](/research/probe-2026-09-04.json). Both are CC0.