Documentation Metrics
I recently had a telephone conversation regarding documentation practices that planted a seed. I’m pretty sure this isn’t a new idea, how could it be, but it’s something that I’ve never seen explicitly stated.
I’m a big fan of static analysis and code metrics in general. The reason I’m a fan is that there’s something in my constitution that likes to see improvement over time. If I think a number correlates strongly enough with software quality, I’ll enjoy seeing that number improve. It’s soothing to me.
What I haven’t seen around is a metric that can be used to measure the quality of a software library’s documentation. Without such a metric, any investment in time towards improving its documentation will never be a sexy thing for me to work on, and I’m probably not alone in these feelings, since most projects have terrible documentation.
I think the # 1 most important thing to measure about a library’s documentation is its web analytics.
Here’s my train of thought: Ultimately documentation is about connecting users to answers. If your page is a promising candidate for solving their problem, then it should receive more quality visits. Anything you do that increases the number and/or quality of those visits is a step in the right direction. If you attempt to game the search engines, it would negatively affect the quantity of hits you’re documentation gets since search engines already take these kinds of things into account.
I’m not exactly sure how the metric will be computed but I’m absolutely sure it’d involve hits, duration of visits and bounce back rate.
I’m imagining a 3rd party documentation analytics embed code that can be used freely on open source projects and that can display on each documentation page a measurement of its “quality” and how its trended over time. It’d allow you to compare two equivalent projects’ documentation, and more importantly tell how much effort is being spent on a projects documentation.
Wouldn’t that be nice? So many open questions. What do you think?
I would guess the best measure is probably in direct user feedback and having a forum to capture frequently asked questions, and complaints. If your docs suck, you will get ample feedback in the forums. This requires no interpretation of fuzzy web analytics that don’t always mean what you think. Over time, you can correct the sucky bits and see less forum questions on that topic.
Some online documentation/help pages ask the user to self-rate: “Did this help?” or “Did this answer your question?” But this is optional…
To use “hits” wouldn’t you have to correlate it to the incoming search string? Kind of like Google does in the way it ties “popular” pages to search words? Maybe just use Google to provide local documentation website search — I think there is a way to get detailed analytics
@Jon Kern
I agree with you that mining forum posts would provide some feedback regarding the quality of the documentation. It’s just quite hard to automatically quantify a change in documentation quality using that approach.
As for the “was this question useful?” stuff, that’s a good idea too, it does however require user interaction and would probably be skewed to the extremes. Also certain forms of documentation don’t lend themselves to that kind of feedback (javadocs for example).
Using “hits” as a metric wouldn’t require you to know what the incoming search string was. You could rely on the search engine to match queries with your pages. It would be helpful to know the search term when deciding how to improve the page’s ranking though.
Thank you for your feedback.