Saturday, July 11, 2009

Evaluations, Yet Again.

Couple more comments on evaluations to follow-up on. First:
"We keep reading that the law doesn't allow CSHOs to be evaluated based on the number of inspections. Where is that in the law? When you quote it to me, read it closely. Doesn't it say RESULTS of inspections? How does that prohibit being evaluated based on productivity -- as opposed to the percent serious, or the size of penalties, etc., etc., etc.?"
OK, here is the text of "Public Law 105-198 - To amend the Occupational Safety and Health Act of 1970:"
Section 8 (h) The Secretary shall not use the results of enforcement activities, such as the number of citations issued or penalties assessed, to evaluate employees directly involved in enforcement activities under this Act or to impose quotas or goals with regard to the results of such activities.
It took a little searching, but here's the Congressional equivalent of a preamble for that law:
H.R. 2877 would conform the law to current practice. It would prohibit the Secretary of Labor from using the results of enforcement activities, such as the number of citations issued or penalties assessed, to evaluate employees directly involved in enforcement under the Occupational Safety and Health Act. It would also prohibit the Secretary from imposing quotas or goals on employees that are based on the results of enforcement activities. The Occupational Safety and Health Administration discontinued using such performance measures and incentives in 1994.
I'm guessing here, and it would be nice if a lawyer type could confirm this, but one of the results of an enforcement activity is an inspection. It's pretty clear that the intent of the law is that we not use inspection numbers as part of our evaluation, even if the language isn't as clear as it could be.

Next, in response to a comment by RT, another commenter left this:
I think that JT's method is used by most supervisors. It's sort of how the performance standards are written, or at least what the supervisor has to put in the narrative to justify any rating other than 'Meets.'

As for the NCFLL wanting objective performance standards, that's a lark. Joe Dear came closest to establishing objective performance standards (i.e. numeric goals) and got throuroughly trounced for it. The NCFLL was the key reason for Kennedy's support of the amendment that prohibits using the number of violations per inspection in the evaluation of CSHOs. That, combined with the number of inspections conducted, may not have been the most perfect evaluation criteria, but at least it was objective.
To which RT responded:
@ Anon: But numbers aren't necessarily objective. Is a CSHO doing a ton of inspections actually effective, writing good citations and focusing on things to get injury rates down, or just a glorified traffic cop writing tickets for the sake of numbers/generating revenue? I think Kennedy's amendment was concerned with government bureaucrats turning CSHOs into the latter.

@Abel: If the problem with performance review is some ADs will play favorites and/or discriminate, then perhaps the solution is to replace the ADs? :) Easier said than done, I know.

Question about the NCFLL - what objective performance elements have they succeeded in implementing? Or proposed without success?
First, sorry RT, it was my post that lead the commenter use JT, and I'm not sure how I got JT stuck in my head.

Second, I agree with the first commenter that there is enough wiggle room in the elements for an AD to adjust the evaluation based on the kind of work a CSHO does, but that also allows ADs who want to help or hurt certain CSHOs the same opportunity.

Third, about what the NCFLL has proposed or been successful with, I'm not sure, I've never been involved in the union activities on a national level and I haven't bothered to keep up with it (I've too much to do already).

Third, changing some ADs might not be a bad idea, but remember "the devil you know..."

Yet another commenter added this:
The problem with using lapse time as a metric is simple. All CSHOs do not do the same type of inspections. In every office, a few of the more qualified staff do more complex inspections. Managers know who these people are and assign them the "beasts." We all know or should at least suspect this.

Emphasizing lapse time without trying to equilibrate for the variable difficulty of inspections creates a disincentive to do complex work and initiates a rush for the bottom. Instead of wanting the difficult cases and being recognized for the effort, folks begin to clamor over the easy stuff.

It's already happening at the end of each FY. That's why any sane analysis of the inspection numbers should tell you that there are bushels of quick and easy inspections/citations being pursued and turned in quickly to lower the lapse rate at the end of the FY.

Incentives work.

IMO, the answer is to create an algorithm that gives different strengths to different types of inspections. Fat/Cats, Wilfull, Jumbo's, Full-shift monitoring, etcetera, could be scored to reflect the increased difficulty level. It's not rocket-science, you wouldn't need mathematicians to do it. Plus, you could incentivize staff to do good, complex work. You can argue over the weighting system, but at least you'd be arguing about real issues instead of assuming something that you already know to be false. All inspections are not created equal.
I certainly agree that there are issues with using lapse time, but if we used an algorithm to weigh inspections, then it seems to me we're crossing the line that Public Law 105-198 set. Using inspection results to evaluate a CSHO means I now have an incentive to increase the penalties to an employer to get a significant case or issue Willful violations. And if I'm going to get credit for full-shift sampling, then that's what I'm going to do every time. I'm going to set up the pumps even if the chemical isn't on the complaint and I know there are no exposures, I have an incentive.

You can see how much discussion lapse time and number of inspections has generated, imagine if this was a complicated issue.

19 comments:

  1. Every manager is evaluated by inspections so that law is useless.

    ReplyDelete
  2. I think, right or wrong, the argument is that those managers aren't "directly" involved in the enforcement action.

    ReplyDelete
  3. For what it's worth, the law is codified at 29 U.S.C. 657(h).

    Food for thought: what's the CSHO's remedy if his or her AD imposes a quota? Can the CSHO file a lawsuit? Or take it up with another agency, kind of like an EEOC claim?

    ReplyDelete
  4. Even more interesting to read the Congressional comments and views on this from back in '98:

    http://thomas.loc.gov/cgi-bin/cpquery/?&sid=cp105pOPWa&refer=&r_n=hr445.105&db_id=105&item=&sel=TOC_1135&


    Over the past three years, the Committee has held numerous hearings on issues surrounding the OSHAct and how OSHA operates. In a series of hearings in the 105th Congress, the Subcommittee on Workforce Protections reviewed the Clinton Administration's plans to `reinvent' the agency and its programs and whether those plans are being successfully implemented.

    The Clinton Administration's effort to `reinvent' OSHA acknowledged that the goals of OSHA were often wrong. In a speech in May 1995 acknowledging the need to reinvent OSHA, President Clinton noted that, `if the government rewards inspectors for writing citations and levying fines more than ensuring safety, there's a chance you could get more citations, more fines, more hassle and no more safety.' 2

    [Footnote]

    [Footnote 2: Remarks by the President on Reinventing Worker Safety Regulation, Stromberg Sheet Metal Works, Inc., Washington, DC, May 16, 1995.]

    The President's recognition that this was the case with OSHA has been echoed repeatedly in testimony before the Committee.

    In March 1995, Ms. Dorothy Strunk, who served as both staff member to the Education and Labor Committee and Acting Assistant Secretary of Labor for Occupational Safety and Health, testified before the Subcommittee on Workforce Protections: 3

    Footnote]

    [Footnote 3: Testimony at a hearing held by the Subcommittee on Workforce Protections, March 8, 1995.]

    Congress, for years, measured OSHA's effectiveness by the number of inspections completed, the number of serious citations issued, the number of criminal cases referred to the Justice Department for prosecution. Are these the appropriate measures to determine the effectiveness of the Act? Or should the question be: `Are hazards in the workplace being abated? Are injury rates being reduced?' That really is the crux of the issue: what is the most effective approach to achieving hazard abatement and injury reduction?

    Again, we are talking about changing long-standing, systemic problems with the agency. Because the agency's success was measured for years by its punitive activity, it has become organized accordingly. If we want, as I think we must, to change the way we measure OSHA's success, we will also need to change the way OSHA is organized--to change the system.

    Other individuals who have had experience with OSHA also have testified about the impact of the agency having had for too long the wrong goals.

    ReplyDelete
  5. To answer the anonymous commenter who asked "Food for thought: what's the CSHO's remedy if his or her AD imposes a quota?" The answer is that evaluations are covered under the union contract and if a CSHO (or any one covered by the contract) has an issue with their evaluation they can file a grievance.

    ReplyDelete
  6. RT back on my May 26 post I asked how we should measure success, so far I haven't heard any good answers. I have no problem reinventing ourselves again, but we need to have goals that are measurable, otherwise how do we know if we're meeting those goals? So what should our goals be and how do we measure them?

    ReplyDelete
  7. Well, the comments to the Act suggest these problems were around ten years ago:

    "OSHA seems to suffer an identity crisis. The Administration calls upon OSHA to reinvent itself. OSHA's response is long on rhetoric and short on substance. During its life, OSHA has accumulated the baggage of an enforcement agency preoccupied with quotas and citations and penalties. Despite hearing assurances to the contrary, OSHA often appears unwilling or unable to embrace fresh concepts and cost-effective measures to enhance workplace safety."

    ...


    "Many employers have complained that OSHA inspectors care less about worker safety than they do about meeting perceived quotas for citations and penalties. While OSHA has not used quotas, it has used citations and penalties as performance measures. I have put a stop to this practice. OSHA's performance is now measured by its success in making safety and health improvements."

    What I get from the comments are:
    1. that it was not passed to protect CSHOs as much as to protect employers from OSHA; and
    2. that Congress was attempting to tell OSHA "you can't do it this way any more" (stop fixating on numbers).

    ReplyDelete
  8. And here's a couple more interesting comments:

    "Many employers have complained that OSHA inspectors care less about worker safety than they do about meeting perceived quotas for citations and penalties. While OSHA has not used quotas, it has used citations and penalties as performance measures. I have put a stop to this practice. OSHA's performance is now measured by its success in making safety and health improvements." (Joe Dear, then Acting Asst. Sec'y)

    ...

    "Despite the change in official policy, however, the testimony of Mr. Schaible, Mr. Nicholas, Mr. Gonzalez, and others from whom the Committee has heard continuing complaints about compliance officers who are concerned that they `will look bad' if they don't issue some citations in the course of an inspection, suggests that the problem is not entirely cured. As Ms. Strunk observed in March 1995 in the testimony quoted above, the practice of judging the agency and its compliance officers based on enforcement numbers is long-standing and systemic. Amending the OSHAct, to make clear that compliance officers may not be evaluated on the basis of such enforcement measures as number of citations issued or penalties assessed will, as the current Assistant Secretary of Labor for Occupational Safety and Health said in endorsing H.R. 2877 on behalf of the Clinton Administration, `help make clear to everyone that OSHA's current policy will not change.'" (Charles Jeffress, then Asst. SOL for OSHA)

    To be continued...

    ReplyDelete
  9. In short, the amendment to the Act tells OSHA to "stop using numbers," but doesn't propose any solutions, either.

    So how do we implement a system that remedies the issue of CSHOs worried about walking away from an inspection empty-handed?

    OSHA could target industries with high injury rates and then, if and when the rates go down, OSHA can proclaim success. But I don't know how you can apply that success down to individual CSHOs. Can you think of any way?

    I like the idea of the "points" system mentioned by another commenter. Such a system, however, could not use the number of inspections, citations, or $ of penalties for any points, lest it run afoul of Section 8(h). What do you award points for? Days in the field, regardless of whether you write up a citation or not? Bonus points for, say, helping a small construction company develop a comprehensive safety program? For giving seminars? For attending seminars?

    ReplyDelete
  10. The last snippet in the legislative commentary:

    "H.R. 2877 simply prohibits the Secretary of Labor from using the results of enforcement activities, such as the number of citations issued or penalties assessed, to evaluate its compliance personnel. The bill also prohibits the setting of goals or quotas with regard to enforcement activities by OSHA. The bill does not prohibit or interfere with other management prerogatives and requirements, such as is involved in ensuring that agency personnel conduct inspections, `give a day's work for a day's pay,' or carry out other agency priorities with regard to targeting of enforcement activities. Similarly, the bill does not preclude the Secretary or her subordinates from reviewing and taking appropriate action with regard to any specific violation, such as where an inspector initially improperly classifies a violation. The bill is directed at prohibiting the practice of evaluating personnel and the agency by enforcement numbers such that the perceived and real purpose of OSHA is punitive against employers rather than corrective and improving safety and health."

    ReplyDelete
  11. I still don't see it. Inspections are not the results of enforcement activity. They ARE enforcement activity. And the comments at the time talked about it being inappropriate to have quotas for "citations and penalties." There is a difference between telling me you expect me to average 50 or 100 or 10 inspections a year and telling me that you expect me to find serious violations in 80 percent of my inspection. In the first case, you're telling me to do my job. In the second case, you're telling me to find violations -- and if I can't find enough, to manufacture them.

    The law says what it says, and the legislative history is consistent with that: It says no quotas for the RESULTS of enforcement activity.

    ReplyDelete
  12. And it's not as though they didn't have a proposal that would have prohibited inspection quotas. The original versino of the bill said `(d) The Secretary shall not establish for any employee within the Occupational Safety and Health Administration (including any regional director, area director, supervisor, or inspector) a quota with respect to the number of inspections conducted, the number of citations issued, or the amount of penalties collected, in accordance with this Act.' If that's what they wanted to do, they could have done it. But they focused on quotas for the results of enforcement, which I still believe excludes inspections themselves. And otherwise why did they make the change?

    ReplyDelete
  13. Bottom line, I think, is that a good manager ought to be able to objectively evaluate the work of their CSHOs. First, they should know who is working, who is slow, and who is just skating by.

    Then come the questions: Whose files don't need constant correcting? Whose citations can be easily defended during an informal due to the amount of documentation? Whose files does SOL have the least problems with? And who do you assign things to when you know it has to be done right?

    The problem with evals comes with the bad managers. Actually, reading through the various posts, the small handful of bad managers in the agency seem to cause an awful lot of problems - CSHOs being assigned inspections they know absolutely nothing about; IHs that seem to think health inspections are a thing of the past; and managers strictly using quotas to measure performance.

    The thing I'm curious about is if this comes from the Area Office level (the AD) or the Regional Office level (the RA). I'd like to work in Abel's or RT's office; I'm not so sure about Kane's or Annie's.

    ReplyDelete
  14. To the last commenter who said, "I'd like to work in Abel's or RT's office; I'm not so sure about Kane's or Annie's."

    All I can say is AMEN!

    ReplyDelete
  15. RE: Abels's comment about how should we measure ourselves... it is how well we reduce injuries, illnesses and deaths. If you don't think we effect that, then you're in the wrong business. There is a region in the country which reduced fatalities by more than 40% in 2008 and then an additional 40% so far in 2009. The managers in this region have a performance element, #6, Special Projects, that measures fatality reduction. This is not rocket science folks-- we measure private industry this way-- we have to do it too.

    When we effect reductions in these areas, everyone is happy. Workers survive their work, workers' comp. and insurance rates go down and we all benefit by making this economy stronger.

    It sounds simplistic and to some extent it is. But it's not a pipe dream. And it's better than chasing numbers. It's being done in a region in OSHA and there's proof in the pudding. I'm not saying we don't make the goals but we get (and give) our rewards from doing the very rewarding work of saving lives and reducing injuries and illnesses.

    I know... and they all lived happily ever after. But imagine if we saw this kind of reduction in all 10 regions in OSHA? Would we be a safer nation? A more productive nation? I think so.

    ReplyDelete
  16. All of the comments posted on the Evaluation Issue are excellent! Nevertheless, I would like to point out the obvious,i.e., the previous commentor indicated that a particular region reduced fatalties by about 40%, and that Managers in this Region utilize performance metric # 6 presumably for the individual evaluations for CSHOs. However, there is a difference between evaluating the agencies performance (regional performance) and that of the individual CSHOs, since to evaluate performance meteric # 6 would seem to require such things as weighting factors, tracking Fatality reductions for the individual companies that a CSHO has worked with, which can have a whole host of confounding variables. For example, after a Compliance CSHO has worked with a company, what assistance did the company receive from Consultation Services. What resources did the company have or allocate in order to obain the necessary changes to their Safety and Health Program, which then translated in improving their safety. Are CSHOs to given all the credit for the work that a company or industry has done to improve S & H, or conversely, are they to be held responsible a down graded in their perfaormance evlaution because they worked in industries or with companies that are complex. As we all know, the metric of reducing fatalities is important with respect to reporting to Congress about our collective success as an agency, as well as for regions,offices, and State Plan States to reprot to OSHA, but it does not tell the whole story. Recently a State Plan State touted its success with respect to reducing fatalities and injuries in both the Residential Construction and Commercial Construction areas due to their "Big Four Emphasis". The problem with this is that this program coincided with the down turn in the economy whereby construction projects markely slowed down. Hence, there could be a confusion a the locus of perception where the reduced moratlity and mobidity rates could be a combination of increased enforcement activity combined with less exposures to hazards related to less work. It should be pointed out that the CSHOs were instructed to conduct limited inspections with the emphasis being on the "number(s)" of inspections and Form 55s they coulde generate. Those that took the time to help employers develop a safety program typically had fewer overall insepction numbers, and were down graded withg respect to their performance. Yes, well all know the balance that we must strike between quality and quatity, between tangibles and intangibles, but since managers are held accountable to the "numbers" in their quarterly reports to Federal OSHA, the "numbers" rein supreme.

    O'Shaman

    ReplyDelete
  17. Sorry for all the spelling errors above, but my dyslexia got the better of me, and I didn't spell check, hence I will do so in the future.

    O'Shaman

    ReplyDelete
  18. And even if you control for all of the variables, the small individual numbers involved make it impossible to determine whether the CSHO made an impact. After all, the worksites that had fatalities this year wil have fewer fatalities next year. Every time. Without fail. And without any intervention. It's simply a matter of statistical probability (both the issue of small numbers and of "regression to the mean"). So be careful about taking credit for fatality decreases. Next year you may have to explain why they went up.

    ReplyDelete
  19. The previous post made an excellent point! Too many folks in power don't understand statistics. Unfortunately, they are the ones that do our performance evaluations based on faulty concepts about the role their individual programs play in reducing fatalities, injuries, and accidents in general. On their behalf though, they have to answer once again to the folks that use raw quantitative numbers (data), and fail to realize the import of the qualitative impact of CSHOs efforts in terms of improving S & H programs so that employers are empowered and employees are empowered to effect positive change, which ultimately translates in reduced fatalities,injuries etc. Further, the employers working with Consultation can markedly improve their S & H programs due to a different dynamic. And, as the previous poster suggested, some employers will improve left to their own devices, and some will simply have the benefit of so-called blind luck,i.e., Statistical Probability. End of dissertation!

    O'Shaman

    ReplyDelete