Methods, Tools & Techniques
Methods, Tools and Techniques are ways of gathering data and collecting the information to learn what changes have happened.
A survey is a brief interview or discussion with individuals about a specific topic. You survey a specific group of people when you ask them a set of questions.
It is common to survey people involved with a service, programme or project. You can survey all the participants in a programme or just a representative sample of a larger group.
Surveys help you understand a group and situation before, during and after it participates in a service or project. For example, a pre-programme survey can let you know the goals, knowledge and needs of participants. Survey them again after each session, and at the end and you can learn what’s changed, perhaps because of your programme. The survey will also show what effect changes to the programme may have had.
Surveying a whole population is very difficult but taking a representative sample – for example door-knocking ten percent of houses in a neighbourhood – may provide a good picture of a community. You can use the results to inform the community, seek people’s views and monitor change over time. Make sure that your survey sample accurately represents the other households that you haven’t surveyed.
Here’s an interesting overview of population sampling.
Three stages in surveying are:
- Planning the process; designing questions and testing.
- Administering the survey.
- Analysing the results.
There’s a range of ways to maximise participation in surveys:
- Offer a prize draw to participants.
- Use clear, simple language.
- Keep survey questions short.
- Use the minimum number of questions you need to build the required evidence.
- Follow-up reminders can help for online and emailed surveys.
LOCAL CASE STUDY
Explore a NZ example of how Te Ora Hou Te Tairāwhiri used surveys to evaluate the effectiveness of their keeping kids safe campaign.
This site explains:
- The different types of surveys, including questionnaires and interviews.
- How you select the survey method that is best for your situation.
- How to design a survey.
- Issues in interviews and some of the advantages and disadvantages of survey methods.
Survey Planning, Design and Testing
Designing a survey is like designing a building – you need to do it properly to get a good result.
How you design questions and how many there are determines whether you’ll get useful results. The order in which you ask them can influence whether people participate and the quality of their responses.
Surveys can ask people to choose from a number of options or keep the responses open-ended. Choosing from options shortens the time it takes to administer and analyse responses but open-ended questions gather more personalised and in-depth information.
It helps to keep surveys as short as possible so respondents participate and give thoughtful answers. You risk their losing interest or becoming impatient if the survey takes too long.
If you ask highly personal or complex questions people may be less willing to offer honest answers. That can be especially so if the person asking the question is not someone they know and trust.
It is always important to test your draft survey with a small sample of people similar to those in the target group. You do this to check that the questions are clear, flow well and provide responses that are as unambiguous as possible.
LOCAL CASE STUDY
Digital Natives Academy sought a robust yet low-cost solution to measuring their effectiveness.
For them, surveys fit the bill.
Administering the Survey
Who conducts the survey and the approach they use is critical to getting good information. People who administer the survey should relate well to those they ask information of. They must demonstrate sensitivity and may need help from professionals who can support respondents if the questions raise feelings, memories or wairua (emotional or spiritual) issues that need attention.
You must inform participants about the survey:
- Who commissioned it.
- What it will be used for.
- Whether any participants will be identifiable.
- Where the information will be stored before, during and after analysis.
- How respondents can have access to the survey results.
- Other matters such as obligations on the surveyor if the respondent discloses information that suggests someone may be unsafe.
It should be easy to administer a survey if the participants are in a programme offered by your organisation. You can survey at the start and end of the programme, or after each session. You could also ask participants to complete a survey six months after the programme ends to see the its longer term impact.
To survey a population, some community groups go door-to-door in a neighbourhood and survey all households. Others take a representative sample of households. You can also survey people on the street, in shopping malls, workplaces or at public events – it’s best always to ask permission from event organisers or property owners well before you start surveying.
Analysing the Results
You can analyse survey results in a range of ways. Analysis may be easy or difficult. It depends on the survey’s complexity and size of sample.
This online guide is for organisations and individuals who want to use a highly scientific approach to survey analysis.
If the survey is amongst a group of only 10 programme participants and asks only five questions, using a scale, then the analysis will be quite fast and easy. If you want to measure the effects of a campaign or project across a whole community your task is greater. You might use an online survey to reach representative opinion then use it to randomly select responses to analyse. A spreadsheet is good to analyse responses from a paper-based survey.
Electronic survey websites and apps like SurveyMonkey and SurveyGizmo are increasingly popular, especially for large-scale surveys. They allow you to process a lot of information quickly and easily. The free versions of these products usually limit the number of questions you can ask and how deeply you can analyse the data. SurveyMonkey provides this useful guide to analysing results.
Advantages of Surveys
An advantage of surveys over other evidence-building methods is that you can design, administer and analyse them quickly and easily throughout the life of a service or project. They are useful for getting a breadth of response from a lot of people though they tend to be less effective for gaining in-depth responses.
Questions asked in surveys tend to be consistent, structured and uniform. They require only short responses – so they can track themes and trends more easily than other formats. Interviews take longer to analyse and observation techniques provide responses that are more open to interpretation.
Most people are used to filling in forms and may find a written survey more comfortable and private than someone interviewing them directly and questioning their responses.
Surveys can reliably represent a large population. When a lot of people answer a survey, the data is more representative of them. Compared with other methods of gathering data, surveys are more accurate.
Surveys are ideal for robust evidence-building because they provide all the participants with a standardised format. Their reliability minimises the researcher/evaluator’s biases.
Limitations of Surveys
If the survey is population or community-based it’s important to select a representative sample to ensure the results reflect the group you are interested in.
People with no or low literacy may find the amount of reading and writing daunting and be put off or be unable to participate. Similarly, cultural groups that prefer face-to-face interaction may find a written survey about important issues impersonal and even offensive.
To ensure a survey’s validity, don’t change the content of the survey or how you administer it once you have started. Being inflexible like this is a survey’s strength as it ensures fairness and consistency.
Deeply personal or controversial issues may not be suitable for a survey unless the people asking the questions have spent enough time building a relationship with the respondents. Expert support may be necessary in the event of significant disclosures about a situation by a respondent and or where someone may be unsafe.
You must standardise survey questions before putting them to the subjects. The researcher must create questions that are general enough to accommodate the average respondent, even if generalised questions may not be as specific to all the participants as they should be.