Conference workshop program: Monday 29 August 2022

>>> DOWNLOAD a printable conference workshop program

Having trouble seeing the full overview table? Switch to mobile version here.
View Sunday conference program here.

WORKSHOP DETAILS

Categories: 
A. Foundational evaluation skills and capabilities 
B. New tools; approaches and ways of thinking   
C. Advanced evaluation topics

Learning to apply the AES First Nations Cultural Safety Framework

presented by Sharon Gollan and Kathleen Stacey    HALF DAY (MORNING) | CATEGORY: A, B

In September 2021, the AES launched the AES First Nations Cultural Safety Framework, developed over a 15-month period via a co-design process with the AES Indigenous Culture and Diversity Committee and representatives from the other main AES Committees.

The purpose of this workshop is to provide participants with guidance and support in beginning to apply the AES First Nations Cultural Safety Framework to their evaluation context. Participants are highly encouraged to have a specific evaluation project in mind that involved Aboriginal and Torres Strait Islander people as project managers, evaluation team members or participants.

NOTE: This is not a cultural safety training workshop. Cultural safety training is usually a two-day learning experience. People who have undertaken cultural safety training will find it a beneficial foundation for this workshop. People who have not undertaken cultural safety training may identify it as a valuable and logical next step in strengthening their capabilities in culturally safe evaluation.

By the end of the workshop, participants will:

• Strengthen their understanding of how principles for culturally safe evaluation can be applied in their evaluation context.
• Develop skills in critical self-reflection in working towards culturally safe evaluation.
• Strengthen their understanding of what it means to be an ally for culturally safe evaluation.

The AES Cultural Safety Framework articulates principles for culturally safe evaluation, and describe how critical self-reflection in relation to evaluators, evaluation roles and responsibilities and evaluation practices can contribute to culturally safe evaluation. Participants will have the opportunity to reflect on and discuss how they consider and apply the principles within their evaluation work. They will consider what cultural accountability means in evaluation, identify how they can take up opportunities and gain practice at being an ally.

This workshop aligns with competencies in the AES Evaluator’s Professional Learning Competency Framework. The identified domains are:

• Domain 1 - Evaluative attitude and professional practice
• Domain 3 - Attention to stakeholders, culture and context
• Domain 6 - Interpersonal skills
• Domain 7 - Evaluation activities

This workshop is aimed at people involved with evaluation in a variety of roles, including commissioners, evaluators, evaluation educators, and managers/staff whose policy, program or project is being evaluated.

About the facilitators

Sharon Gollan, Sharon Gollan & Associates, has strong affinity with and is an active community member of the Ngarrindjeri group in South Australia. She has over 35 years of experience in the public sector in a range of community services and management positions primarily focused on creating better services for Aboriginal peoples, followed by eight years in academic teaching and research before focusing all her time on running her training and consultancy business, as well as contributing to better life outcomes for Aboriginal peoples through Boards, State Committees and Working Groups.

Sharon has invited beyond... to work with her since 1999 in developing and delivering training programs focused on cultural respect and safety as part of organisational change projects and reconciliation action plans (RAPs). They also work together on a regular basis to develop and/or evaluate health, education, family and community service programs at state and national levels.

Kathleen Stacey, beyond...(Kathleen Stacey & Associates) Pty Ltd, is the Managing Director and Principal Consultant at beyond... She spent her formative working years within the public sector and academia, before establishing and expanding beyond... into its current form. The company conducts consultancy, evaluation, research and training work in human services fields, with a specialist focus in health, education, youth, early childhood, mental health, and family and community support services. It has developed a strong reputation for culturally respectful work in Aboriginal programs and organisations and has worked consistently and collegially with Aboriginal consultants across a range of projects since beyond…’s inception in 2000. 

> back to overview > register

Applying the AES First Nations Cultural Safety Framework in practice 

presented by Sharon Gollan, Kathleen Stacey, Doyen Radcliffe, Ginibi Robinson    HALF DAY (AFTERNOON) | CATEGORY: A, B

In September 2021, the AES launched the AES First Nations Cultural Safety Framework, developed over a 15-month period via a co-design process with the AES Indigenous Culture and Diversity Committee and representatives from the other main AES Committees. This has been supported by providing a series of ‘Learning to apply the AES First Nations Cultural Safety Framework’ foundation workshops - see the AES online events calendar for dates: https://www.aes.asn.au/evaluation-learning/professional-learning-events.

The purpose of this workshop is to build on the foundational workshop by providing participants with an opportunity to learn from and discuss examples of applying the concepts and approach described in the AES First Nations Cultural Safety Framework.

NOTE: This is not a cultural safety training workshop. Cultural safety training is usually a two-day learning experience. People who have undertaken cultural safety training will find it a beneficial foundation for this workshop. People who have not undertaken cultural safety training may identify it as a valuable and logical next step in strengthening their capabilities in culturally safe evaluation.

Pre-requisite: Attendance at the foundational ‘Learning to apply the AES First Nations Cultural Safety Framework’ workshop in the morning is a pre-requisite.

By the end of the workshop, participants will have reflected on and discussed actual examples of culturally safe evaluation that will: 

• Enhance their understanding of how the principles of culturally safe evaluation can be applied in their evaluation context.
• Strengthen their skills in applying critical self-reflection to their own evaluation contexts.
• Enhance their understanding of what it means to be an ally for culturally safe evaluation.

This workshop will be run as a panel session, where two different evaluation teams will share an evaluation example that strove to be consistent with the Framework and facilitate a discussion with participants to explore the application of the Framework. Participants will get the opportunity to walk some of the journey of each evaluation, addressing scenarios or challenges that emerged along the way, and practice applying the concepts and approach in the Framework to identify how they would respond if faced with a similar scenario or challenge in a future evaluation.

This workshop aligns with competencies in the AES Evaluator’s Professional Learning Competency Framework. The identified domains are:

• Domain 1 - Evaluative attitude and professional practice
• Domain 3 - Attention to stakeholders, culture and context
• Domain 6 - Interpersonal skills
• Domain 7 - Evaluation activities

This workshop is aimed at people involved with evaluation in a variety of roles, including commissioners, evaluators, evaluation educators, and managers/staff whose policy, program or project is being evaluated.

About the facilitators

See biographies above

> back to overview > register

Creating impact through systems-led evaluation and design

presented by Matt Healey   FULL DAY  |  CATEGORY: B

Our existence is interwoven with the complex contexts we inhabit. We both influence and are influenced by the systems around us, and it is increasingly acknowledged the benefits that incorporating systems thinking into our practice can bring. However, in many ways the ‘practice’ of systems is largely inaccessible in our day-to-day work. While we acknowledge that ‘thinking systemically’ and ‘being systemic’ is beneficial, the next question is often ‘how?’

The purpose of this workshop is to step attendees through the application of systems thinking techniques to the design and evaluation of an intervention to address a complex problem. Starting with an exploration of our complex problem, over the course of the day participants will:

• Learn and practice techniques to map complex systems
• Apply different approaches to identifying ‘where’ to intervene in the system, and how to justify it
• Be introduced to the pros and cons of different design techniques (eg. co-design)
• Develop an evaluation framework for understanding systems change, including implementation requirements.

This workshop will demonstrate the benefits of framing ‘how’ we understand systems as a blended approach with the design and evaluation of interventions to support change. 

This workshop has been designed from the bottom-up to be highly practical and hands on. Participants will be stepped through systems mapping, design and evaluation techniques throughout the day as they are applied to an example complex problem designed by the presenter. In addition to the core activities which will be undertaken in small groups, there will also be individual reflective activities designed to support participants to consider how these skills can apply to their own work.

Given the interwoven nature of session content, Competency 1: Evaluative Attitude and Professional Practice, Competency 4: Research Methods and Systematic Inquiry and contributes to Competency 5, in being able to choose appropriate methods are addressed.

About the facilitator:

Matt Healey is a Senior Consultant and Co-Founder of First Person Consulting. He works on projects that span health, social justice and environmental sectors that seek to address complex problems. This includes family violence, homelessness, prevention, social innovation, e-waste and the circular economy (among others).

In addition to evaluation, Matt also has a strong interest in approaches to the design of projects, programs and services. This ranges from human-centred design, co-design, the use of evidence and working with clients to figure out 'what works' for them. In recent times this has extended to systems-based approaches to understanding complex problems, and what we might do to address these.

> back to overview > register

Values literacy in evaluation

presented by Keryn Hassall     HALF DAY (MORNING)  |  CATEGORY: A, B, C

There is a growing interest in engaging with values in evaluation, to better understand what matters to people and communities in order to make appropriate evaluative judgements. But what are values, and how can you make sense of values when planning and conducting evaluation?

Working with values is not as simple as learning a new technique. First, you need a good understanding of values and how values relate to policies and programs. ‘Values literacy’ is a starting point for developing this knowledge: “the cultivation of a set of epistemic skills and attitudes in recognizing, analyzing, and interpreting values.”

This workshop is designed to help participants to: 

• Understand what values are (and are not), drawing on the current understanding of values in social psychology and philosophy
• Recognise different kinds of values influencing a policy, program or evaluation
• Consider ways to elicit and discuss values when planning an evaluation, to clarify ‘what matters’ for the context
• Understand the range of options for engaging with values that are available in the evaluation literature.

About the facilitator:

Keryn Hassall is an evaluator working across policy areas. She believes that evaluators can enrich their evaluation practice through learning about philosophy of social science. Her consultancy work is primarily formative and process evaluation. She uses ideas from philosophy, organisation theory, and public administration research to help organisations evaluate and improve programs. 

> back to overview > register

The centrality of Indigenous Data Sovereignty and Indigenous Data Governance in the evaluation of Indigenous programs

presented by Maggie Walter, Bhiamie Williamson, Skye Trudgett and Bobby Maher   HALF DAY (AFTERNOON) |  CATEGORY: B, C

IDSov and IDGov are terms increasingly used within Aboriginal and Torres Strait Islander communities and organisations, and within government when Indigenous evaluation projects are mooted or planned. These linked concepts emerged as a response to the long described deficiencies and problems of data collection on or about Indigenous peoples and used across Anglo-colonised settler nation states. This creates a pattern of extractive data practices often combined with weaponisation of data against Indigenous Peoples. The broad use of the terms IDSov and IDGov do not always translate to a broad understanding of the concepts. More critically lack of a nuanced understanding can quickly lead to the continuation of poor quality data practices magnified by them being incorrectly labelled as IDSov or IDGov. To directly address this ever present risk, this presentation and interactive session begins with an exercise in how NOT to implement IDSov in evaluation. We will then focus on the Aboriginal and Torres Strait Islander IDSov and IDGov principles, and the how to operationalise the principles in your evaluation practice to ensure an Indigenous-led approach to IDSov.  

About the facilitators:

Maggie Walter (PhD, FASSA) is Palawa, a member of the Briggs Aboriginal family in Lutruwita/Tasmania. Maggie is a Distinguished Professor of Sociology Emerita at the University of Tasmania and was appointed a Commissioner with the Yoorrook Justice Commission in 2021. Maggie is a founding member of the Indigenous Data Sovereignty group in Australia (Maiam Nayri Wingara) and an executive member of the Global Indigenous Data Alliance (GIDA).

Bhiamie Williamson is a Euahlayi man from north-west NSW and south-west Qld with family ties to north-west Qld. He holds a Bachelor of Arts (Hons) from the Australian National University, a Masters of Arts in Indigenous Governance from the University of Victoria in British Columbia and a Continuing Education Certificate in Indigenous Governance from the University of Arizona. Bhiamie's fields of professional and academic experience include Indigenous land and water management, Indigenous youth, Indigenous governance, and Indigenous data sovereignty. He is a current member of the Mayi Kuwayu Data Governance Committee and the ACT Bushfire Council. Bhiamie is currently enrolled in a PhD at the Australian National University. His PhD investigates Indigenous Men and Masculinities.

Skye Trudgett is a Gamilaroi researcher who has contributed to numerous evaluations and research projects. Skye has a passion for Indigenous Data Sovereignty and amplifying the voice of First Nations peoples in impact measurement and evaluation. Skye has recently completed her PhD at UNSW and leads Kowa, a First Nations organisation creating systemic change through innovative learning partnerships. 

Bobby Maher is Yamatji woman, her ancestral links are to the Kija people, Pilbara regions and Noongar Nations. Bobby is a PhD candidate and Research Associate at the National Centre for Epidemiology and Population Health, Australian National University; her research has a focus on collective capability in Indigenist evaluation practice in Australia. Bobby has completed a MPhil in Applied Epidemiology and a Bachelor of Applied Science (Indigenous Australian Research) (Honours), Curtin University. She has experience in quantitative, qualitative, and community-based participatory research, including evaluation. Bobby is a member of the Maiam nayri Wingara Indigenous Data Sovereignty Collective and the Global Indigenous Data Alliance (GIDA).

> back to overview > register

Introduction to data visualization – theory

presented by Mark Griffin (supported by David Cowen)    FULL DAY  |  CATEGORY: A

Data visualization is an important technique for evaluators as they seek to convey the patterns present within their datasets to their readers. It is known that readers will typically spend between 3 to 8 seconds looking at each statistical plot, and so we need to present our data in such a way that it makes the most impact within this short time window.

This workshop focusses on the design of effective data visualizations. This workshop focusses on the theory, and a companion workshop focusses on the use of appropriate software for creating these visualizations.

Specific content (“Mark’s Principles of Visualization”):

• The difference between Exploratory and Explanatory Data Visualization
• Key Message – the one most important message you want to convey to readers
• Call to Action – what do you want readers to do in response to seeing your plot
• Instant Recognition – will readers be able to see the pattern you highlight instantly
• Reduce Clutter (Signal to Noise Ratio) – remove every bit of ink on the page that does not support your Key Message
• Be Purposeful – make conscious decisions about every aspect of your data visualization, do not just use software defaults
• Beauty – beauty in visualization promotes trust, enjoyment and further interaction with readers

The workshop will discuss theory and present case studies to illustrate good and poor data visualization. The workshop is suitable for evaluators without prior knowledge and aligns with Domain 7 – Evaluation Activities of the AES Evaluators' Professional Learning Competency Framework.

About the facilitator:

Dr Mark Griffin holds five degrees in the areas of statistics, mathematics, and computer engineering. He is a Casual Lecturer in the School of Business, University of Queensland and has previously presented approximately 100 two-day and 40 five-day workshops in statistics across Australia.

> back to overview > register

Creative evaluation and engagement

presented by Kate McKegg, Nan Wehipeihana, A Rafael Johnson, Nora Murphy Johnson    FULL DAY  |   CATEGORY: B

The purpose of this workshop is to introduce participants to a way of embodying a radically new evaluation purpose. Content will be guided by a set of principles and move through four phases: align, learn, adapt, and embody. The facilitators will introduce participants to the roots, guiding principles, methods, and essential strategies of Creative Evaluation and Engagement (CE&E). Participants will be supported in learning how to practice purpose-driven, principles guided evaluation in a supportive, generous and constructive way. It will introduce participants to the steps and tools to practice evaluation in ways that are self-aware as well as aware of the interconnectedness and wholeness of living systems.

The workshop is directly relevant to all the domains of the AES Evaluators' Professional Learning Competency Framework. It will combine face to-face and online facilitation with four of the world’s leading evaluation guides and facilitators. It will be engaging, creative, highly participatory, and full of inspiration. This workshop is for those with at least some understanding of evaluation and systems change, but we welcome all levels. There are no other prerequisites other than being prepared to bring your whole self, be curious and creative. 

About the facilitators:

Kate McKegg is the director of The Knowledge Institute Ltd (www.knowledgeinstitute.co.nz ) and a member of the Kinnect Group (www.kinnect.co.nz), as well as an indigenous led collective Tuakana Teina, based in the Waikato region of New Zealand. She is also a co-founder, along with Nan Wehipeihana and Nora Murphy of the Developmental Evaluation Institute (https://developmental-evaluation.org/about), and a founding member and past Convenor of the Aotearoa New Zealand Evaluation Association (ANZEA).  

Kate is co-editor of New Zealand’s only evaluation text, Evaluating Policy and Practice, a New Zealand Reader (2003). She is also co-editor (along with Michael Quinn Patton and Nan Wehipeihana) of the book Developmental Evaluation: Real World Applications, Adapted Tools, Questions Answered, Emergent Issues, Lessons Learned, and Essential Principles, Guildford Press, New York, (2015). 

Seeing the potential in a complexity aware evaluation practice to support change, Kate has been drawn to developmental evaluation and other creative forms of evaluation practice because of her deep commitment to social and environmental justice and equity. She has worked alongside many people in complex settings who are innovating to create systems change and seen the possibilities that a different kind of evaluative practice can bring. 

Nan Wehipeihana is the director of Weaving Insights (www.weavinginsights.co.nz) and a member of the Kinnect Group (www.kinnect.co.nz). She is also a co-founder of the Developmental Evaluation Institute (https://developmental-evaluation.org/about) along with Kate McKegg and Nora Murphy. Nan is a founding member of the Aotearoa New Zealand Evaluation Association (ANZEA) and Ma Te Rae, Māori Evaluation Association – the first Indigenous Evaluation Association. Nan tribal affiliations are to Ngāti Tukorehe and Ngāti Raukawa, north of Wellington and to Ngāti Porou and Te Whānau-ā-Apanui on the East Coast of New Zealand.

Nan is a Fellow of the Australian Evaluation Society. Nan, Kate McKegg, and colleagues were the recipients of the 2013 AES Best Evaluation Policy and Systems framework for the Developmental Evaluation of He Oranga Poutama for Sport New Zealand. Nan is a co-editor (along with Michael Quinn Patton and Kate McKegg) of the book Developmental Evaluation: Real World Applications, Adapted Tools, Questions Answered, Emergent Issues, Lessons Learned, and Essential Principles, Guildford Press, New York, (2015). 

Nan specializes in research and evaluation with a focus on Māori and building evaluation capacity with tribes and Māori organisations to evidence outcomes including cultural outcomes. By bringing the voices and views of Māori to government and funders, she aims to offer insight into Māori values and perspective for use in government, business, and community contexts.

A. Rafael Johnson is the Vice-President of Inspire to Change, where he uses the methodologies of the arts to understand systems, organizations, and programs. His fiction and essays have appeared in Temenos Journal, AEA365, Callaloo, Kweli Journal, African American Review, and the anthology Excavating Honesty: An Anthology of Rage and Hope in America. Andy is an adjunct faculty member at the Minneapolis College of Art and Design, where he currently teaches Creative Analytics with Nora Murphy Johnson. Andy holds an MFA in Creative Writing from The University of Alabama and is a fellow at Kimbilio Fiction. His newest book Creative Evaluation and Engagement: The Essentials (co-authored with Nora Murphy Johnson) revives Michael Quinn Patton's 1982 classic Creative Evaluation. Essentials positions the arts as an essential way of knowing and communicating, and prepares changemakers to collect inspiring data and build a body of evidence that can make the world more whole, just, and beautiful.

Nora Murphy Johnson is the President of Inspire to Change and its principal investigator. Nora believes that all systems of people and institutions are connected and that all parts of the system need to be strong and healthy. Nora encourages clients and stakeholders to think outside of our boxes and disciplines and create a coherent shared vision for something greater than what exists now. Evaluation can be an integral part of working towards this vision. Nora works towards understanding (1) how principles-focused, developmental evaluation can be used for systems change and social justice, (2) ways to create a coherent and shared vision that allows for contextualized learning and adaptation, and (3) how to best engage people in useful evaluations that inform and inspire. Nora is best known for her publications Nine guiding principles to help youth overcome homelessness: A principles-focused developmental evaluation (Developmental Evaluation Exemplars, 2015) and Connecting Individual and Societal Change (Stanford Social Innovation Review, 2020). Her newest book Creative Evaluation and Engagement: The Essentials (co-authored with A. Rafael Johnson) revives Michael Quinn Patton's 1982 classic Creative Evaluation. Essentials positions the arts as an essential way of knowing and communicating, and prepares changemakers to collect inspiring data and build a body of evidence that can make the world more whole, just, and beautiful. She holds a PhD in Evaluation Studies from the University of Minnesota.

Nora will join part of the workshop via Zoom.

> back to overview > register