How to Choose Meaningful Metrics – Federal Leadership and Professional Development Seminar Series

How to Choose Meaningful Metrics – Federal Leadership and Professional Development Seminar Series

Articles Blog


our next presentation is measuring
performance and how to choose meaningful metrics so we have Boris Oratia here
from General Services Administration who’s the senior advisor there at the
GSA’s office of shared solutions and performance improvement he works with
agency and cross agency teams to design and deliver collaborative initiatives
that advance progress towards solving complex cross-cutting
federal government challenges as part of the PMA for example he is currently
supporting a cross-agency team that is working to improve the American public’s
experience with federal services please join me in welcoming Boris thanks so
much Thank You Kim and it’s gray seen so many
familiar faces here as we’ve been having a really great conversation on the whole
vision of strategy and looking at goals and and kind of pulling it all that
together with Adam we had that Chris we had Leonor and we had David – not sure
if they’re here but even having David and the panel reminded me it’s this is a
really a stellar event I was really bad stellar of course but but I really I
have enjoyed working with many of you in this room and what I want to do the
first thing I want to say is as I share with you so I’m coming with my abuse and
having done a lot of work this is not an official you say do you per se but I
just wanted to make sure that we’re just clear as we go about here so we’re
sharing a lot of best practices and I just want to make sure that this becomes
something that’s really practical for you so as as I have a few minutes here
and really before lunch is me so I’m very aware of that so my job is to
increase your appetite for meaningful metrics do you like that one Robin okay
good all right all right thank you yeah thank you here on Wednesdays and
Thursdays yeah so this business is really it can get complicated and one of
the things in my career is really try to simplify things making them easy to
understand we’ve got a lot of different stakeholders representing here we’re
talking headquarters we talk agency and then also Bureau level and
and I want to say a lot of yours those of you who are joining on the Adobe
Connect on the webinar we want you to keep you engaged as well keep asking
questions and and so I just want to first of all say that the objected I
have here the few minutes that I’m going to be sharing with you so two things I
want to do the first thing is I want to inspire you to really go out there
capture and share the good things that you’re doing in your programs every one
of you and even just looking around the room and and the folks in the in the
webinar really gives me a great inspiration myself to just do the things
that we do because we’re all at the end of they were trying to really improve
the way we deliver services to to the citizens and so the other thing is I
want to give you a process as well the it’s gonna be easy to understand it’s
born not on theory there’s a little logic model in that that’s in here and
some of you here that you know exactly what I’m talking about all the ones here
it’s like what are you talking about but it’s really looking at cause and effect
so I’m gonna talk a little bit about that but it’s really more about giving
you something practical so I want to do those two things they inspire you and
then give you some practical things that you can start doing tomorrow there’s a
lot of good stuff here in the afternoon so don’t get super inspired start
working on your metrics right away so wait till at least tomorrow so you know
before we get too far into this the first thing one would ask is what we
measure so we’ve been talking a lot about strategy with looking at how to
pull things together and and coming from so I spend a lot of time doing CPI Lean
Six process improvement and there’s always you know ways to improve but I
think you got to look at the plans that we just talked about how we’re gonna
track that we’re actually on track and how are we going to make sure that we’re
delivering good quality work and the last question there is are we focusing
on the right thing so these are the types of questions that we want to ask
before you start with anything is a good idea to us why and so when it comes to
meaning I’ve got a lot of slice here so I’m going to pass in a few of them but I
just wanted to make sure so don’t be alarmed by the sheer number of slice I
wanted to give you something that you could actually use but when it
comes to what makes a metric meaningful you know there’s a lot we can say and
and again there’s books are written about this topic but I really boil it
down and this is experience working with cross agency priority goals working with
some of these some of you here in the room I’ve had the pleasure to work with
your teams and do some facilitating as well Robin yeah recently but but it’s
really about linking to desire outcomes so if you remember one thing from what
I’m saying I want you to remember that statement this is really what we’re
trying to get to here the other things that are important as well but I want to
make sure that you also think about being easy to understand appropriate for
the audience and incentivizing the right behavior so I heard Lenora talk about
yes we want to make sure that we’re measuring results but let’s not forget
about behaviors you know what is it that we want to be able to see at the end of
the day so that’s really important and then the other thing is we don’t want to
forget its quality do we have quality metrics having done this for awhile this
doesn’t happen overnight so quality is a process so you’re going to sometimes
collect data get some metrics and they don’t necessarily if reflect what’s
really happening and so you’re gonna have to do a couple cycles till you get
there so don’t don’t be alarmed it’s not gonna be great quality at the top but
it’s something that comes with time so don’t give up so there are other things
that will make them a metric meaningful I’m sure you can think of all the things
but I want to make this real personal right now so I wanna I want you to pass
for a minute here it’s not mental silence but I want you to think about
your role right now in a program or initiative and I just want you to think
about minute to think about two or three things that you really care about things
that you’re measuring right now that really make a difference to you I know
we’re coming to the end of a performance cycle maybe you’re thinking about your
performance plan and how am I gonna do with the conversation but maybe there’s
that but think about one or two metrics that you really care about right now and
I want you to think about why so there’s a table here if you’re in the webinar
just take a min or so to to kind of think about and jot down one or two
metrics because someone I use that in some of the other things I’m going to
talk about so I’m going to stop talking for
I’ll give you about a minute to write one or two things that you’re measuring
right now that you really care about for a program or an initiative go okay and
as you’re writing that also think about why do you care I mean why should we
care we asked about that earlier but why do you care about that metric what makes
it relevant to you okay and the last thing in this metric is regardless of
whether they say you start about you’re gonna have different roles you’re gonna
be the person producing the data you’re going to be the evaluator you’re gonna
be using in some way or the decision-maker your consumer of that
data so there’s always different roles that you play or perhaps you don’t play
any of those were also thinking about that that’s important too all right you
got your metric in hand great okay so the last thing that I said is as you
thought about that metric or two or three that you care about why you care
about them then the last thing is are you producing the data to measure that
metric or someone else doing that or perhaps you’re the decision-maker that’s
gonna use that data to make some decisions okay so what what I want to
say is so we talked a little bit about the why and the meaningful when we talk
about meaningful when it comes right down to it again in this business for a
while it comes down to three things when you’re looking at measures and I don’t
know if Adam was here maybe he’ll agree I don’t know you might throw a couple
other things here but but I really look at it as we’re always gonna be answering
questions from state holder’s and we’re dealing with all
kinds of stakeholders so the organization that I work for at GSA so
we support both cross agency priority goals teams we do some work with agency
priority goal teams as well but then also we support secretive councils
that’s a lot of what my office does the pic boom one of them I see a lot of
fellow pic members and and friends here which is really good to see but we’re
always asking that you know are really answering the right questions and are we
giving the stakeholders the things that they’re looking for are we managing our
work that’s the other thing that metrics really help us do and then are we
providing transparency so we talked about that as well there’s all the
things but if I if I were to say three things this is what I would say so
moving from there what I want to do next is I want to lay here a process that I’m
gonna say straight on through so we’ve done this a number of times and just
even looking around the room I’ve worked with a couple of you doing this process
teaching it but also applying it and a lot of the the perspective I’m gonna
share here is the facilitator perspective so a lot of the work that I
do at GSA so I help facilitate a lot of the discussions that happen both we
senior say and then cross agency as well so I want to give you that perspective
because what I hope that you will do is this something that you can take and
work with your teams so laying this out in a way that a facilitator would think
about I think it would be helpful for you to take it and export it to where
you’re going to use it so we don’t get too far in this conversation without
talking about outcomes we’ve been talking about that already and that
really becomes that first stopping point here there’s a set of questions here I’m
not gonna read them all to you but I wanted to just put in there because
these are very questions have been thought out carefully to kind of guide
this conversation and you’re gonna start with your stakeholders if you don’t
really have a good grasp on the on the stakeholders I recommend that you do
have some kind of stakeholder analysis and and this is a basic project
management 101 but it’s also very basic to one we’re talking about strategy
goals vision mission and then metrics thinking about the outcomes that matter
to them again you may need to go hey shocker you may need to untuck to
some stakeholders on asking what matters to you and and that’s some of the things
that Lenore and Chris we’re talking about you get out there and you talk to
people find out what is it that that makes them tick thing you’re going to
try to figure out we talked about behaviors as well so that’s another key
point that I want to make today the behavior piece is really important
because a question I would ask what are the behaviors that we can observe that
indicate these outcomes are actually happening so that’s a really important
thing and I want to illustrate with a couple examples so we do that and notice
we haven’t touched metrics yet so we’re gonna do that conversation so this is
the thoughtful part of it and and when you do this I recommend doing it as a
team again we talked about whether you do it by yourself or you’re working with
a team and then you enter the metrics conversation with a solid foundation so
when you get into that second block you’re really looking at you know how do
we validate these observations again a roundabout maybe touchy-feely way of
dealing with it but I found doing this for a number of years that really works
when you start talking about behaviors and data you know how what data do you
need to validate things are happening and so you go from there and then you’re
going to flow into the the last block which is planning and this is where you
you basically say okay how we’re going to define this metric very important a
definition that makes sense to everybody that everybody can agree to but it also
something that we will be able to get the data and actually consistently get
the data and be able to track our performance so we’re going to do that in
the planning phase so one one quote that I like to use when I’m talking about
this is I like from a quote from Andy Stanley they’re the author and it is
Direction not intention determines destination not Michael this is a nice
friendly so Direction not intention determines destination and I really
think that really applies in the business that we have been talking about
here today so when we start with this process I want unpack it a little bit
because I want to take us through again how you will be facilitating this
measures development process so in the the first stage we’re looking very
closely our stakeholders this is a topic perhaps for
another day stakeholder analysis is kind of near and dear to my heart so we do a
lot of that and the things that we work with in our team trying to figure out
who is who and there is a pecking order as we know we’re looking at stakeholders
which ones are we gonna focus in this next cycle and so you want to have those
hard conversations sometimes but then you’re gonna be asking what matters to
them you know what other things that are important to them and what are the
outcomes that they expect you to get so when I’m facilitating this conversation
I’m asking teams we’ve got a lot of different outcomes we can put up on the
walls and we got stickies when I saw this white walls I almost grabbed a
sharpie and start writing because in GSA we have a lot of white walls so I I will
try not to do that and not get you in trouble here we go go to town yeah so so
we’re gonna be asking a lot of those questions but we’re gonna synthesize
outcome statements this is perhaps the most important part of this process
because this is really the thing that we’re trying to get to and I want to
define outcomes a little more I know we have that conversation often times but
we’re gonna have that that process of synthesis and typically I like to work
with teams and getting down to three there’s no scientific method here but I
found that much more than that tends to make it harder for people to focus 3 is
about right too many to feel from that sometimes it makes it not as as good for
looking at a broad range of things so 3 is about the right number not magic but
that I found that works well after we have this conversation on outcomes I
first want to differentiate when we’re talking about outputs we’ve been talking
a little bit about this already we really look in that scale we’re looking
at volume and the things that we’re delivering we’re not necessarily looking
at the quality or the the things that make us special or really is it getting
to what we’re trying to accomplish that’s really when we talk about
outcomes again we have a lot of conversation on this with this I found
this a good way to differentiate things so we’ll describe it in outcomes they
intend the result of the program or the project or the activity and it’s going
to help us assess the success of the program we’re actually getting to where
we’re trying to get so employ more practical way to show
I’ve got a couple examples here well let’s take this event here by the way I
wanna recognize game for putting this together this is really cool because we
do this a lot in DSA and the more we do this the better it is for all of us so
thank you so value proposition not something I was going to get into here
too much but I’m gonna just give you in kind of a nickel tour of that the value
proposition is really asking the question what is our value add to this
customer and sometimes you’ll have that so there’s a whole conversation you
could have in value proposition and that really ties more into developing your
mission vision and goals but typically value proposition is asking that
question what is the value add if my organization disappear will they miss us
okay so with this example here so this conference if we’re looking at an
activity so the activity itself is we’re holding this conference this seminar
this event an output of that will be the number of people that are here in the
room people that are in the chat in the webinar but then also we’re looking at
the topics we cover so we’re measuring the scale and all those things are
important but if I want to look at the outcome side of this matrix
perhaps I’m looking for the number of programs initiatives or projects that
benefit from information that was gathered at this conference so that’s a
little more of an outcome base where I’m looking what is the value of pulling
these people together perhaps the number of partnerships that were born I’ve made
a lot of friends in events like this and then we wouldn’t wind up working
together so that’s more of an outcome base and so I wanted to give you that
perspective as we’re thinking about and don’t over thinking now
but that’s just a kind of a simple way to just kind of differentiate the two so
the next part of the conversation really comes to behaviors and I really like
this one because when you have some great outcome statements on the world
you got this big state we use this big stickers which I love you got this great
outcome statements everybody I agree now you you want to give it some context so
what are we really talking about so I love the question is what are the
observable behaviors that indicate outcome is happening and a little
straight with a couple examples in a minute so we have that second
conversation we still haven’t talked about metrics here keep that in mind so
we have that conversation also all behaviors and then we’re trying to get
to again the providing that sort of meaning giving that meaning that that
outcomes some meaning so I want to use a a couple examples so work with a couple
of cross agency priority goals things actually this were from the previous
administration what we were looking at the security clearance and reform
process and with the team cross agency team from different agencies we
generated three outcome statements here efficiency quality effectiveness and so
the worst might look I know that’s pretty broad doesn’t mean anything well
it means a lot to the people that were in the room that we’re working with so
young Robin is is life in there yeah but it means a lot to the people are working
on this if you look at effectiveness so let’s look at that one there the
security suitability and credentialing processes reduce a person incidents from
in from insider so that’s an outcome statement again this is not an agency
perspective it’s across agency perspective so we’re looking at the
community as a whole and so when I do a little more of a deep
dive on that into observable behavior so we had that conversation now we were
looking at three different observable behavior so I wanted to highlight the
first the SSC community as a whole can show how well it identifies hostile
actors and ballets good actors so that gets to give in a little more context
we’ve got a couple other statements here but that’s where you again you begin to
us what we want to be able to observe so now there may be more simple example if
you got kids are going back to school it’s been a long time for me all right
but anyway a lot of you have young kids are going out yeah not dating myself
here and so when you may be a desired outcome is healthy bodies through
nutritious lunch right so when you pack the carrot sticks in the in the box and
also all behaviors they ate the carrot sticks and you say wait a minute maybe
they just gave in to the friends and they got a Snickers bar right they did
some trading that kind of thing but but that’s a little bit of the logic here so
we’re trying to again this is what I want to accomplish what is it I can
observe you’re trying to look things that you could actually observe another
example so we work with cross-agency grouped I was looking at
the SES core and he was looking at metrics for for measuring the strength
of the SES core and this is again initiative from the previous
administration and we were looking at some outcome statements so I want to
focus on accountability the last one there so one of the things that we the
team came up with is we want to be able to hold senior execs accountable for
achieving results in support support mission success so that kind of sounds
broad means a lot to the people that were in the room as we were building
these statements and so as we unpack that one into the observable behavior
here we have accountability we’re looking at a couple of things so there
is these there is a regular and consistent feedback process okay so I
have got something that I can observe not only in one agency but actually can
observe that in across the agency so we had that conversation again before we
enter the the realm of the metric so the next piece is here this is very
sequential one thing I want to pause here and say that when you’re doing this
in real life you’re gonna iterate so I didn’t want up again having done a lot
of process improvement over the years I’d like to draw little process Maps but
it’s really cyclical and you’re gonna eat away at every step so over here you
ask the question alright great so I know the outcomes I know the behaviors so
what data do I need to validate these behaviors are actually happening and so
that’s where then you begin to look at some metric ideas perhaps you’re already
measuring things you want to look at your existing metrics for sure but then
you’re going to start generating ideas so this becomes a brainstorming process
and I like to use the final idea here where you’re really looking for what are
going to be that shortlist so one of the things I heard Lenora say is you’re
looking for small amounts of metrics that would really tell the story capture
the story that you’re trying to tell and so as you’re going through this final
process there is different activities you can use for brainstorming and one
one activity that I like is the the could can and will it’s a no brain
storming activity but we are said first so we will break it up into three time
slots so we’ve got our outcomes we got out of several behaviors you put those
on the wall with your team so first question you’re asked for a few
minutes okay what could you measure this is anything goes this I don’t care about
the data whether I can get it I don’t care how hard it’s gonna be to measure
this thing let’s just think outside the box
and that’s the good question so you spend a couple minutes stickies again
flying on the walls here the next question is what can you measure so you
bring them down the funnel a little bit more here’s more like okay I know this
is gonna be hard to do but I know we can do it actually we have the people we
have systems we can upgrade and then you get down to you know let’s talk turkey
here what is it yeah close to luncheon maybe shouldn’t use the word turkey here
but I know sorry so the the what will you measure gets down to okay what are
those narrow types of things that we really want to focus on and so what
you’re going to for here is quantities any of the view run brainstorming we’re
looking for quantities and then you’re going to have some things to look at now
after you’ve done that activity I like to think about this process again as
you’re going to be iterating you’re going to be engaging stakeholders and
you’re going to be asking typically three areas here that I call the 3m s 3m
s which is really meaningful measurable and movable so within each of these so
these are the types of things that are going to help you narrow down the list
what is it that would really want to focus in measuring here the first thing
I said when we started this conversation does it link to desire outcomes super
important is it gonna give us information to to make decisions right
and so you won’t so on and so forth and and as you going through this process
again this is iterative but you’re trying to narrow your your your field
down you also going to look at your existing metrics and perhaps some of
your system metrics are doing this but maybe they’re not doing this and if the
metric that I asked you to think about before and and if you’re having thinking
about okay what are the outcomes so we’re trying to get to what our server
behavior so you may wanna question that metric if it’s not really working for
you right okay so so this is what we will do to sort of iterate so I’m going
to go back to the two examples the first one where again we’re looking for the
that metric ideas for the security suitability and credentialing process so
one of the things that team came up with is the SSC process effectiveness so that
means absolutely nothing by looking at that but it means a lot to that team
because I’m going to show you a couple steps on how to actually define this
matrix then we have our investigation effectiveness reporting effectiveness so
each one of these went through another process of definition to really nail it
down what are we talking about and then for the SES strengthening initially if
we had a couple of them one of the metric ideas was let’s use what federal
employee viewpoint survey is already producing so we have a a a rave rich
data source has multiple years and there’s a couple of questions there’s a
link exactly to where we’re trying to do so we were able to leverage something
that was already been done and then you see the other metric ideas again so this
is very quickly I kind of run you through the process here a facilitator
process again that I’ve seen work very effectively particularly when we’re
working with complex types of things so when you get to this point the question
comes up alright so I’m there I’ve got a good idea my outcomes are I’m looking at
my of several behaviors the things I want to be able to see and now I got
some ideas on metrics maybe I have five metrics maybe I have ten depending on
the number of outcomes before you get to that step I wanted to go back to the
slide because when you have your set of metrics then you’re gonna go back to
your outcome statements remember we had three outcome statements let’s say that
you were looking at a program that ended up you’re gonna go back eight do these
metrics really linked to that outcome the behaviors would give us some context
I’m not so worried about this point about that because that was just a step
to get to the metric ideas but now the outcomes are important so you want to
look at that and test them you know that first question does it link to the
desire outcomes really keep asking that question as you go so you get to this
step and and normally by the time you get here one thing that I would
recommend is before you spend the time to define a metric take the time to
really sit through because this takes time this is laborious and it’s gonna
take some time so you want to get down to the the point where you’re you know
not a whole lot of metrics but you’re gonna spend it the effort to really
define this metric and to think about the measurement approach what is the
measurement approach so we’re gonna take so these are templates that we use all
the time in fact this past week I was facilitating a couple groups at GSA that
we’re doing exactly this so I’ve seen it work at a fairly low level as far as a
fairly complex multi agency level so the typical questions that we’re asking here
is okay how do we define this metric so when we’re saying let’s say that there
was surprised that I was working at GSA we’re looking at the onboarding
experience for new employees and so we’re one of the metrics we’re looking
at is the number of employees I get it function in PC on day one okay how many
of you showed up somebody’s laughing yeah I got a PC I can get to work
excellent so we’re measuring that so we define you know what are you talking
about so this is really the ability for an employee to get a computer on the
first day and for them to be able to at least to email at a minimum so we we put
that definition in there what’s the purpose house is linked to desire
outcomes desire outcome is we want to make the new employee experience I’m
boarding the new employee onboarding experience a pleasant one so that was
one of the outcomes so that links to that for sure our owner and audience
here we’re looking at who owns that I heard the term ownership in the previous
conversation this is really important and I think Chris was talking about that
too when you think about this there’s gonna be an ownership piece and the more
you have a clean line of who actually it’s the one that owns this metric the
easier it’s gonna be H is absolutely right the easier it is going to be to
get to some results I seem teams I really take this serious
really do some great things particularly with the desired outcome so it really
took the time to define this I rockem’s so then the last question in the
definition template is is it relevant how is it going to help me make
decisions so if you have trouble answering any of these questions as
you’re developing the metric then you might need to go back to the drawing
board do I have to write metric before you even go any further here so that’s
the beauty of this so you’re gonna use that process you may not have all the
right people necessarily in the room you’re going to then engage other folks
as you going along so do you dive a little deeper once you kind of have an
idea of how you’re going to define this metric and now you start asking deeper
questions such as okay where’s the data are gonna come from
who’s got the data how we’re gonna collect this so data collection that was
the whole thing so many of you who work in in some kind of performance
improvement initiative or you’ve done it in the past you you know that usually
you’re gonna have to come up with a metric somehow you’re gonna be able to
say I’m the current stays here future stays here so I need to be able to see
if we got better so this is really about sometimes having a data collection plan
if I don’t have the data how we’re gonna collect there so there’s a lot that
happens here then the next thing is about how you’re going to disaggregate
what are the groupings of this data that are meaningful to the people that you’re
working with then you’re going into the more granular things like how how we’re
gonna report this how often we’re gonna do it so it’s a conversation that you’re
gonna have as you’re trying to define the metric and the last but not least is
what are some of the barriers that I could have in trying to actually pull
this thing together pull it off in a way that’s gonna help us if those barriers
are insurmountable perhaps you go back to the drawing board or you look at some
of the other ideas because for this year even though it’s a great idea links to
the desired outcomes I might need to postpone that one maybe two till I have
conditions that will allow me to do that sometimes that’s the decision so this is
again a process are you gonna use – to build metrics I wanted to really give
you the thinking process more than really getting into the nuts and bolts
here because that’s the part I’m really interested in you taken away from here so when we define accountability make
sure I understand so observe all behaviors are really there to give
context to the outcome so accountability if we’re talking about what we have here
in terms of who owns the metric we’re really looking at who is going to be the
main person that’s going to have control on what happens in that the inputs that
go into the metro it’s just really who has the control and
sometimes the answer to that is multiple stakeholders there’s multiple players I
have control and then you saw have to allocate who what is my piece right so
sometimes it’s really not the whole thing but what is my contribution make
sense okay so with all the things that I talked
about here so that the things I want you to walk away with first of all is you
start with a focus in understanding your desire outcomes where is that you’re
going you’re gonna think about your behaviors the ones that you want to be
able to observe to help you give context to that outcome and then you’re going to
iterate obviously and this truly is a team process something that you’re going
to do with a group of people I don’t recommend going it alone sometimes we do
that and sometimes that’s the only option I may not be realistic sometimes
our teams are so small that we end up you know that’s real life right for us
all of us but then you wanna irritate you bring all the players in the in the
mix here and so that’s really the key points that I have for for a message
here on the process this is a much extended process that we’ve done in the
past but I just wanted to kind of consolidate it to two small piece here
so um I have a couple of questions for you to think about one I entertained do
we have time for questions which I think probably a lot of people struggle with
how long do you collect information before it is considered valid and not
just part of your baseline okay so that’s a good point don’t really have a
real clear answer other than when so so let me back up a little bit so there is
target setting I didn’t talk about target setting so when you have your
metric you’re trying to set a baseline me if this is a matter you’ve never done
before let’s just say this is something new so you’re trying to set a baseline
you’re going to collect some historic you look at some historical data perhaps
you start collecting some data at some point you’re going to say okay I’m I’m
ready to start collecting data from this point on and and I would say maybe a
rule of thumb is when you feel like you have a system that’s gonna give you some
reliable data and that might take some iteration
because once you have a baseline you also going to have a desired future
state and somewhere in that line there’s going to be a target that you’re going
for that’s not necessarily going to be on the desired future state and so once
you know what that baseline is and you say okay I’ve got some reliable way at
least collecting some quality data I’m gonna say that with the caveat even in
your best efforts you may not necessarily get the quality data he
thought you were gonna get because sometimes you’re getting it from sources
that you have no control on that’s sound familiar right so you’re gonna get the
data you do your best and then at some point you guys okay I have a fairly
reliable process for collecting the data so that’s a long answer I don’t I can’t
get to it all right so I do have a couple of questions to think about I
want to open it up to you all to to see if you have any thoughts or questions or
insights as we were just talking about this here yeah causality and what I mean
by that is maybe we achieve the outcome that we were after and we did have
metrics that linked to that outcome but what I can’t I don’t have evidence
of is is what I did or what the project manager did are they is that what caused
that outcome or was it something completely extraneous to what we were
doing okay so that gets into a little bit on the evaluation piece so where I’m
collecting the data I feel like I have decent quality data and I feel like it’s
a good metric it links to the outcome and so as I’m looking at the information
that’s coming now evaluation is gonna help me and that’s topic so we’re gonna
get into this afternoon but we’re gonna talk about K what does it mean and what
is it really telling me because ultimately you’re trying to get to some
outcome and it’s not really helping me get there so one thing that we do is
root cause analysis sometimes you can kind of geeky a little bit with the
process improvement stuff so glucose analysis is really what is the root
cause if there is something that’s happening when I’m looking at that data
that caused that to happen so I would say typically that falls more into
evaluation piece I’m not sure if anybody else wants to add anything
that sure I think there’s about and on the side the other things that we’re
doing or leading to democracy in a country right but the whole point of
coming up with a logic model is to say we think that these activities lead to
these outputs which lead to these outcomes and we just sort of have to own
the fact that there might not be some causal linkage there that we’ll never be
able to prove but we always advise our bureaus to at least monitor and evaluate
according to that logic because that’s the whole point I think someone earlier
it said you’re sort of testing you have a hypothesis you make a plan and you’re
testing the hypothesis that’s exactly what they’re doing and so we just say
sometimes you can’t but the idea is that you’re you go to the evaluation
sometimes to see if all those things do add up to the bigger picture but the
first step is documenting the fact that you think these things add up to the
bigger picture right that’s a good point and and the even though I didn’t talk a
lot about logic model in it is sometimes when I use that term people’s or what
are you talking about but it’s really some of the logic really is in this
process and that is you know again inputs the activities the outputs
different levels of outcomes short term mid term long term and then impacts and
you’re looking at how you know cause and effect actually that is good for
progress monitoring and even assessing and the level of outcome sort of change
that you have accomplished based on the project or intervention and we we talk
of Sallah t it’s always important a comparison group of some kind the logic
model is going to take you more to a process evaluation or outcome evaluation
and the monitoring or even performance measurement but not necessarily
causality for any kind of causality you all we have to have a comparison it can
be a control group and it can be a say my experimental design and I need is
based on the comparison with something similar and then justice Alcott’s and
even if you cannot see behaviors you can look at least attitudes for instance I
reflect democracy or not in an area where they have not received the kind of
project or intervention okay good point let’s see we have one here from the
webinar we have a question what metrics can you use to validate a process for
example risk management our line money spent to mitigate number of red yellow
and green risk okay so I will answer the question with a question what’s the
desire outcome so even in your er M your enterprise risk management and think
about what is the desire outcome that you’re going after and then let guide
the decision because you could you can collect all of those things and they
might give you a picture of what was happening there might be there something
else that you’re not collecting so that that would be my question first of all
you know what is it that I’m trying to get to that specific I mean we can be
very general but the beauty of outcomes is we’re really trying to get more
specific what is it that we’re trying to accomplish let that be the answer so
what I wanted to leave you with some questions to think about as you’re doing
this work and and by the way this is how we tell the story of the great things
are happening so the thing I get passionate about this I know sometimes I
school is over when you say metrics but I get passionate about this because this
is how you tell the story of all the good things that you’re doing so so it’s
important that that as you doing that think about you know are you including
your stakeholders and some of your saying yep we definitely are some of you
are thinking you know we could do better with that so I’m gonna challenge you to
think about that both internal and external when you’re thinking about what
you’re going to measure the next question is are the majority of people
on your team agree that you’re really measuring the right things and you may
or may not know this and this goes back to communicating and and really asking
people and even down to the frontline you know you’re measuring things but are
we really measuring the right stuff you know ask asking that question from the
from time to time not just because you’re measuring it
and then is this is a an iterative process or are you revising your matrix
from time to time and are they are they becoming easier to work with are they
more clear as you’re updating and upgrading your matrix as you go along
and then think about what is a great metric that you’re currently using in
your program and and is this something that I want to replicate as you’re
thinking about that so that metric that I asked you to write
down when you started now you got some things to test it against and I want you
to do that and that’s my goal so if you perform metric out of this talk I feel
good all right so I go back to my question all right
did did help you get to the this that typically we measure more than we need
to that’s what I’ve been my experience we tend to overdo metrics to be honest a
few places where I seen that what we don’t have enough with the typically I
mean that shouldn’t generalize most of the times I see as measuring more than
we really need to so I really comes down to is it helping you answer your
stakeholder questions is it really helping you manage the work and is it
really giving you the transparency that you need so I would say that should
guide you if you have I’d say more than 10 metrics I’m going to throw a number
out there that are that are linked to one specific project maybe you have too
many so I’m just using a blanket statement here but it tested that way is
it really answering that the right questions and this is something that is
really going to help me make decisions if you don’t get good answers to those
questions then perhaps you want a question that metric I realize sometimes
you don’t have a choice you’ve been told you’re gonna measure this right so there
is sometimes some of that tension that you’re going to run into the room okay
thank you so much for us and if we have other questions for Boris just email
them to me thank you very much I wanted to say we
have a lot of resources in the pic website pic resources there is a couple
of play books we have some quick guys that the play books are particularly
helpful we spend a lot of time curating collecting practices from you all
actually and I want you to feel free to share and then if you want to contact
any of us we have an email inbox we got a pic stuff at gsa.gov and i will get to
me and they will get to my colleagues as well and we certainly help folks in
thinking through this this whole process so pic staff at gsa.gov will get me and
others thank you

Leave a Reply

Your email address will not be published. Required fields are marked *