Buying

How to verify a WYSIWYG vendor's feature claims

Every editor's comparison page says “included”. A feature list is a marketing artifact until you can exercise it. Here is how to check any vendor's claims in an evaluation — including ours — in about half an hour.

9 min read

Every editor’s comparison page is a column of green ticks. Ours included. The ticks are cheap to type and expensive to verify, which is why they drift from the product — not usually through dishonesty, but through a documentation page written from a roadmap and never revisited.

We know this because we found it in our own site twice. Our accessibility statement said “No keyboard trap: users can always Esc out” and “toggle states via aria-pressed”. Neither was true. Separately, our cloud site advertised conversion endpoints the API had never implemented. Both were written ahead of the work and nothing connected the claim to the code.

So here is how to check any vendor’s claims during an evaluation, in about half an hour, without their help.

Step 1: probe the runtime, not the docs

Almost every editor exposes its capabilities as methods on an instance or entries in a plugin registry. Load the vendor’s own demo page, open the console, and ask the editor directly what it can do:

// Whatever the vendor's constructor is:
const ed = new RichTextEditor(document.createElement("div"), { toolbar: "basic" });

setTimeout(() => {
  console.log({
    footnotes:   typeof ed.getFootnotes,
    pdf:         typeof ed.exportToPdf,
    docx:        typeof ed.getDocxBlob,
    pageView:    typeof ed.setPageView,
    rtl:         typeof ed.setBaseDirection,
    trackChanges: !!ed.trackedChanges,
  });
}, 2000);

"undefined" next to a feature on the comparison page is the whole answer. This takes five minutes and is more informative than an hour on a sales call, because a method either exists or it does not.

We ran exactly this against our own 22 “Included” rows: 24 capabilities probed, 24 present. That is the number you should be asking any vendor to produce — not a tick, a probe you can re-run.

Step 2: check what actually ships

A method on a demo page can come from a build that is not the one you download. Fetch the distributed bundle and look for the implementation:

curl -s https://vendor.example/editor/plugins/all_plugins.js \
  | grep -c "Plugin_Footnotes"

Zero means the feature is not in the file you would be shipping, whatever the demo does. This also catches a subtler problem: a capability that exists in the vendor’s source but is excluded from the packaged build for the tier you are buying.

While you are in there, check the same bundle for outbound network calls. A metered editor has to count you somehow, and the counting code is in the file:

grep -oE "https://[a-z0-9.-]+" all_plugins.js | sort -u

We wrote up the fuller version of that check in does your editor phone home.

Step 3: the claims most often written ahead of the code

Some claims are far likelier to be aspirational than others. In rough order of risk:

Step 4: read the silences

What a comparison page omits is often more informative than what it claims. Two patterns worth noticing:

A missing “known gaps” section. Every real product has gaps. A page with no limitations listed is not a product without limitations; it is a page written by marketing without engineering review. A vendor who publishes their gaps has usually measured them.

Ties recorded as wins.Check whether the comparison marks anything as a tie or a loss. If every row favours the vendor, the page is an advertisement rather than a comparison, and the individual rows deserve correspondingly less trust. Our own page records ties — TinyMCE’s formatting marks and CKEditor’s show-blocks are free-core features, and CKEditor’s automatic text transformation means our autocorrect is parity, not a win.

Step 5: ask them to pin it

The deeper problem is not that a claim was false once; it is that nothing prevents it drifting again. A feature can be removed from a bundle, or a docs page can be written before the sprint that implements it, and no ordinary test notices — a test suite checks that code behaves, not that the marketing site is true.

That is fixable, and it is a fair thing to ask a vendor whether they do:

// Every "Included" row on the comparison page must name a symbol
// that exists in the shipped bundle, or the build fails.
const claimed = [...vsData.matchAll(/label: "([^"]+)",\s*rte: "Included/g)].map(m => m[1]);

const unmapped = claimed.filter(label => !(label in CLAIM_EVIDENCE));
assert.deepEqual(unmapped, [],
  "These rows claim Included but have no evidence mapping. Add one — or stop claiming it.");

for (const [label, symbol] of Object.entries(CLAIM_EVIDENCE)) {
  assert.ok(bundle.includes(symbol), `${label} is claimed but not implemented in the shipped bundle`);
}

We added this to our own suite after the accessibility statement embarrassed us, and tested it the only way worth testing such a thing: by adding a fabricated row for a feature that does not exist and confirming the build failed and named it.

It is not sophisticated. It is roughly sixty lines. But it converts “we try to keep the site accurate” into a property the build enforces, and that is the difference between a claim you can check and a claim you have to believe.

A note on fairness

Almost every inaccurate claim we have found — in our own material and elsewhere — came from a page written before or after the code rather than from anyone trying to mislead. Treat a wrong tick as a process failure, not a character failure, and ask what the vendor changed so it will not recur.

“We fixed the page” is a weaker answer than “we fixed the page and added a check that fails the build next time.” The second one is the one worth paying for.

RichTextEditor is a perpetual-licence JavaScript editor — one purchase, self-hosted, no metered editor loads. Download the evaluation or see how it compares.