We undertook a study of validity and reliability of WCAG 2.0 and found that an 80% target for agreement is not attainable, when audits are conducted without communication between evaluators. Even with experienced evaluators the error rate is relatively high; and further, untrained accessibility auditors -be they developers or quality testers from other domains- do much worse than this. Read the full published text via ACM Author-izer Open Access on the publications page.

Scatter plot of difficulty vs correctness, sensitivity, and F-measure for experienced evaluators.

You can also cite us using the bibtex at the bottom of this post. But what does this really mean? Well it means that it’s going to be very difficult for the 80% agreement WCAG 2 requires of expert evaluators. Now this isn’t the deal breaker it may seem – as we do see agreement at the 70% mark; and so agreement does still happen – however a strict adherence to the standard cannot be achieved – at least in our strict experimental work. It maybe that a post evaluation round table discussion would push up the agreement, it maybe that a collaborative round table evaluation would push up the agreement. We don’t know, as there is no empirical work out there to support this.

Giorgio Brajnik, Yeliz Yesilada, Simon Harper (2012). Is accessibility conformance an elusive property? A study of validity and reliability of WCAG 2.0 ACM Trans. Access. Comput. DOI: 10.1145/2141943.2141946

Citation

@article{Harper2012hgy,
Abstract = {The Web Content Accessibility Guidelines (WCAG) 2.0 separate testing into both “Machine” and “Human” audits; and further classify “Human Testability” into “Reliably Human Testable” and “Not Reliably Testable”; it is human testability that is the focus of this paper. We wanted to investigate the likelihood that “at least 80% of knowledgeable human evaluators would agree on the conclusion” of an accessibility audit, and therefore understand the percentage of success criteria that could be described as reliably human testable, and those that could not. In this case, we recruited twenty-five experienced evaluators to audit four pages for WCAG 2.0 conformance. These pages were chosen to differ in layout, complexity, and accessibility support, thereby creating a small but variable sample.

We found that an 80% agreement between experienced evaluators almost never occurred and that the average agreement was at the 70–75% mark, while the error rate was around 29%. Further, trained—but novice—evaluators performing the same audits exhibited the same agreement to that of our more experienced ones, but a reduction on validity of 6–13% ; the validity that an untrained user would attain can only be a conjecture. Expertise appears to improve (by 19%) the ability to avoid false positives. Finally, pooling the results of two independent experienced evaluators would be the best option, capturing at most 76% of the true problems and producing only 24% of false positives. Any other independent combination of audits would achieve worse results.

This means that an 80% target for agreement, when audits are conducted without communication between evaluators, is not attainable, even with experienced evaluators, when working on pages similar to the ones used in this experiment; that the error rate even for experienced evaluators is relatively high and further, that untrained accessibility auditors be they developers or quality testers from other domains, would do much worse than this.},
Acmid = {2141946},
Address = {New York, NY, USA},
Articleno = {8},
Author = {Brajnik, Giorgio and Yesilada, Yeliz and Harper, Simon},
Authorizer = {http://dl.acm.org/authorize?6630377},
Crossref = {http://dx.doi.org/10.1145/2141943.2141946},
Date-Added = {2012-04-24 09:00:13 +0100},
Date-Modified = {2012-08-21 17:47:39 +0100},
Doi = {10.1145/2141943.2141946},
Issn = {1936-7228},
Issue_Date = {March 2012},
Journal = {ACM Trans. Access. Comput.},
Keywords = {Accessibility Guidelines, Evaluation, Web accessibility},
Month = mar,
Number = {2},
Numpages = {28},
Pages = {8:1–8:28},
Publisher = {ACM},
Title = {Is accessibility conformance an elusive property? A study of validity and reliability of WCAG 2.0},
Url = {http://www.simonharper.info/publications/Harper2012hgy.pdf},
Volume = {4},
Year = {2012},
Bdsk-Url-1 = {http://doi.acm.org/10.1145/2141943.2141946},
Bdsk-Url-2 = {http://dx.doi.org/10.1145/2141943.2141946}}