[Research notes] How to use on-line data on September 11th?

How to use on-line data on September 11th ?
(Application of P.F.Lazarsfeld’s ‘elaboration model’ in just-in time research.)
Hynek Jerabek
Charles University, Prague, Czech Republic
 

 
RESEARCH QUESTIONS:

  1. Can we find some general, culturally independent, repeated spontaneous reactions to the exceptional events of September 11th?
  2. Can we validate some culturally independent (repeated in different parts of the world) pattern of communication behaviour in the hours after the September 11th events?

DATA:
We have voluntary responses from a non-representative population – 2578 on-line questionnaires, filled in mostly by frequent Internet users, the highly educated, younger and most active members of their societies (2090 Czechs and 488 other nationalities around Europe and the world).
Just-in-time RESEARCH DESIGN:
The just-in-time research on September 11th had to be carried out within a very short period of time. It was essential for the respondents to be able to recall as precisely as possible what they did right after the event, their opinions, and their spontaneous reactions at that time. We wanted to determine what their opinions and reactions were like before they were influenced by the mass media, which understandably were commenting on the event in the hours and days after it occurred and that had a certain influence on the views of the population. The 41 questions were concerned mostly with the initial reactions of people and their communication behaviour in the hours after the September 11th events.

This speed with which the research had to be done influenced its research design. There was no time to organise a mass collection of data using the traditional methods of F2F or CAPI research. There were no financial resources prepared for use in such data collection. From this it is obvious that there was a lack of resources and time for selecting a representative sample of respondents.

On September 11th we invented a questionnaire for the purpose of an on-line international communications research. Just two days after the September 11th events, the Network Media Service (Eva Veisová and her colleagues) began an on-line collection of data in three languages: English, German and Czech. The recruitment of respondents was made through electronic newspapers, press and through a snowballing e-mail distribution of web addresses of questionnaires.

The original aim of the researchers was a very ambitious one. We wanted to address ‘people around the world‘, including respondents from Asia, Africa, and the Middle East. This is apparent from the identification questions in the questionnaire, which were formulated with this aim in mind, for example the way in which the level of education is coded as ‘education in years’, characteristics ascertaining religious affiliation (Christian, Muslim, Jew, Buddhist, Atheist, other), language, and continent. It is also apparent from the closed question about the emotions the respondents felt, where we asked, particularly with a view to respondents in the Middle East, or other countries, what are the most appropriate emotions. Alongside the options ‘I was surprised’, ‘I was astonished’, ‘I was shocked’, ‘I was afraid’ and ‘I was angry’, we also included the options ‘I was happy’, ‘I was satisfied’, ‘I was glad’, ‘I was pleased’, which we assumed would certainly not be selected by inhabitants from western Europe or the United States. They were clearly intended for future comparative analyses among various cultures and civilisations on our planet. However, unfortunately we received no response from respondents from countries and continents outside Europe and North America. We can recommend to future researchers who find themselves in a similar situation to have contacts prepared in advance to websites in distant countries to turn to with requests to post the questionnaires. Unfortunately we didn’t have these contacts
DATA COLLECTION, EXPENSES AND FIRST PRESENTATIONS

The research design as a whole had to work with almost no budget. And so the costs of the research involved the work of a team of enthusiastic researchers, several translations of a short questionnaire, and the fees for posting the questionnaire on websites. The speed required in order to capture people’s immediate reactions was also reflected in the data collection. Most data was collected in the first 14 days. The results of the just-in-time research were presented on the international field very quickly. The first paper written up on the results of our research on September 11th was presented very soon after at a special roundtable on ‘September 11th’ at the WAPOR Annual Conference in Rome, Italy, on September 21, 2001, e.g. 10 days after the event. First publication (in English and in Czech) has been published in November 2001 in Prague (Jerabek-Veisova 2001). As a result the research as a whole was not just very quick but also very inexpensive.

However, it was necessary to think about a method of analysing the data and a model for presenting the results that would guarantee the validity of the scientific findings. The sample of respondents in the research was certainly not representative.

RESPONDENTS – WHO ANSWERED?
The final on-line data file contains 2578 completed questionnaires used in the analysis. The respondents included 2090 Czechs and 488 foreign nationals from literally all over the world: 88 Slovaks, 143 Germans, 104 inhabitants of other European countries, 81 citizens of the United States, Canada, Great Britain and Australia (53 of them from US & CAN), and 72 people from Latin America, Asia, and the Middle East. 59% of the respondents were men and 41% were women. The gender composition is balanced in all language and nationality groups.

METHODOLOGY:
WHAT IS OUR METHODOLOGICAL PROBLEM?
How can these data be used to provide some more generalizable statements about the first reactions and communication behaviour of Czech, European, and American populations?
SPECIFICATION OF THE METHODOLOGICAL PROBLEM:
We are aware of the problems related to Internet surveys, as pointed out, for example, by Janet Hoek, P. Gendall, and B. Healey. In their view, the primary question surrounding Internet surveys is: ‘… the extent to which email or web-based surveys can produce estimates that are generalisable to the wider public…’ [Hoek, Gendall & Healey 2001: p.2.] According to these authors on-line surveys of the general public ‘are likely to result in highly skewed samples with a strong bias towards younger, better educated and higher income males.’ [Hoek, Gendall & Healey 2001: p.6]
George Terhanian and his colleagues from Harris Interactive applied an approach to Internet-Based Surveys of Non-probability Samples using a combination of a large non-probability sample drawn from voluntary responses on the Internet and a substantially smaller probability, and thus representative, sample of responses. [Terhanian-Black 1999, Terhanian at al. 2001] With the use of ‘propensity score adjustment‘, the results relating to sub-groups of respondents of Internet audiences are re-weighed on the basis of characteristics that influence the respondent’s probability of being an on-line respondent.
We were faced in our comparisons with an even more difficult task than Terhanian and his colleagues. In ‘Just in time’ research it was impossible for us to carry out a comparative survey of probability sample respondents. Nevertheless, we were and are still interested in finding a way to use the acquired data for statements whose validity is applicable beyond the ‘Internet population‘. Our Strategy has been based on the ‘Survey Analysis’ approach.
THE ANALYTICAL STRATEGY FOR OUR METHODOLOGICAL APPROACH:
We strove to apply Paul Lazarsfeld’s methodological principles of elaboration [Lazarsfeld 1955 {orig.1946}, Lazarsfeld – Kendall 1950, Rosenberg 1964, Zeisel 1985, Babbie 1994] in a situation of incomplete data representation. We must defend, explain and validate that our separate non-representative groups of volunteer respondents represent in some respect the other parts of the population.
WE USED LAZARSFELD’S PRINCIPLE OF REPLICATION:
If the statement is repeatedly valid, under the control of all relevant external variables, for all different cultural settings, we can confirm our statement as universal in relation to our scope of inquiry.
Our analysis was aimed at tracing the universal spontaneous reaction, opinions and models of communication behaviour repeated in many cultural and linguistic environments. Any valid statement was conditioned by repetition in different settings and we strove to formulate it as a finding only in the case of the recurrence of identical statements in sub-populations, which mutually and significantly differ from one another.
FINDINGS:
If the relationship recurred among people from different countries (Germany, US & Canada, Czech Republic), from different parts of the world (Europe, America), we would have grounds for concluding that the original relationship was a genuine and general one. [Compare: Babbie 1994: p.395]
1. PEOPLE DISCUSSED THE SEPTEMBER 11TH EVENTS LATE INTO THE NIGHT
Altogether 99% of the Internet population discussed the events in the USA with friends or relatives. This figure is valid for all territorial parts of our on-line voluntary sample (98% ‘– 100%). Perhaps only a few of us can recall an event that registered as much attention from people throughout the world. Our main hypothesis was confirmed – the events of September 11th became an important stimulus to conversation for people everywhere.
How long did people continue to discuss these events?
The comparative data from our on-line sample show that we can distinguish two groups of respondents. 90% of Czechs, Europeans and Latino-Americans talked with their close ones or with friends for at least one hour. Voluntary on-line respondents from US+Canada and Asia+Near East discussed the event significantly less – only about 75% of them for one hour or longer. The distribution of short (contra time-consuming) discussions probably reveals some cultural differences. In America, the percentage of people among the most active and highly educated population (both men and women) who discussed these events for 10’ – 59’, was significantly higher (many times) than in the rest of the world.
In spite of these cultural differences we can conclude that interpersonal communication – conversations about this international event – was thus without a doubt one of the main activities of that day, afternoon and evening, in many families throughout the world.
2. WHICH WORD COULD DESCRIBE WHAT HAPPENED ON SEPT 11?
We used (and are interpreting) in our questionnaire two types of questions: open-ended and closed questions. There are two types of reactions to the events of September 11th. The first group of respondents understand the rational meaning of the question as ‘asking for a description’. The second group understand the emotional meaning of the question as ‘asking for sentiments and feelings’.
The ‘open-ended’ question permitted respondents to use their own words, and to use many more different words. People mostly reacted emotionally, and therefore we found in all sub-samples a predominance of words with an emotional content. In the spontaneous reactions of our on-line respondents we counted, without significant territorial differences, 81.6% of words with emotional content: tragedy, catastrophe, angry, shock, afraid, bestiality, madmen, barbarians, apocalypse, Armageddon, horror, sci-fi, unbelievable. The other words (18.4%) have more of a factual content, such as: ‘terrorism’, ‘war’, ‘attack’, ‘global attack’, ‘revenge’, ‘expectation’; or referred to uncertainty: ‘hard to say’, ‘don’t know’, ‘other’. Generally speaking, in the spontaneous reflection of the September 11th events emotional reactions prevailed in all tested territorial populations.
TABLE 1: WHICH WORD COULD DESCRIBE WHAT HAPPENED ON SEPTEMBER 11TH ?
CLOSED QUESTION DESIGN –   ITEM EVALUATION

tab
If we asked for specific words, for an evaluation of each of them on the scale between
‘the most appropriate’ and ‘the least appropriate’, we found the word ‘terrorism’ in first place among the ‘most appropriate words’ (in all sub-samples) (See table 1). The next words from this group of ‘rational’ meanings (attack, war attack, war) were found in third place. The word ‘tragedy’ was in second place (also in all cases) and the word ‘disaster’ was found in fourth (for most sub-samples). Word ‘assassination’ was very frequent in Europe, especially in German subsample, where it took a third position.
3. WHAT EMOTIONS DID PEOPLE EXPERIENCE ON SEPTEMBER 11TH?
In our on-line research the most frequent answer to the questions: ‘How can you best express the emotions you felt when you first received the information? If we ask you directly now and offer a few options, which would you pick as the most appropriate?’, was ‘I was shocked’. 81% of respondents chose this statement as the most suitable description of their emotional state, and 90% of respondents indicated it as ‘most appropriate’ or ‘appropriate’. Other options frequently indicated were: ‘astonished’ (48% – the most appropriate statement) and ‘surprised’ (44% – the most appropriate statement). All groups of respondents indicated positive emotions as an absolutely inappropriate expression of emotion. No one felt happy, no one was glad, satisfied, or pleased. An almost identical outcome was found in all our sub-samples. We can see the results in Figure 1.
FIGURE 1:COMPARISON OF AVERAGES FOR STATEMENTS EXPRESSING EMOTIONAL STATES
figure1

Note: A higher average indicates a more appropriate description of the emotional state; 4 = the most appropriate; 0 = the least appropriate

CONCLUSIONS:
What could be our methodological conclusion from our ‘Just in time’ communication research? We know very well that our experience with the analysis and interpretation of non-representative data could be an unrepeatable and truly exceptional case. The unique, exceptional situation of the September 11th events provided an extraordinary interplay of circumstances in which social scientists had the opportunity, and also the obligation, to study some universal, global tendencies and some inter-culturally repeated relations and facts, which cannot usually be studied. It was for this reason that we did it. We hope that the uniqueness of the September 11th situation can, in some respect, vindicate the really extraordinary sequence of non-traditional deviations from the mainstream of public opinion methodology that we used.

We looked for recurring patterns of spontaneous reactions to the event of September 11th across the different cultures and civilisations the respondents came from. However, this was possible only to the extent to which the sample of respondents allowed. Naturally we also looked for differences within the studied sample of respondents. We found differences by education. However, we only discovered them after our research was repeated by Robert Chung on a representative sample of 520 CATI respondents of from the Cantonese-speaking Chinese population in Hong Kong. (Chung, R., Jerabek, H. & Veisova, Eva 2002) Our sample was too homogenous from the perspective of education. The ‘internet population’ in the Czech Republic and in Europe at that time was characterised by above-average education levels. We did not detect differences by age because our respondents were on the whole just young people. We found differences between men and women. They recurred in all the studied sub-samples of compared groups of countries. Sometimes perfectly, sometimes partially. In the case where fear was acknowledged there was an interesting difference in this respect between inhabitants of the United States and Europeans. In the United States the difference in the acknowledgement of fear between men and women to a ‘real threat’ was smaller than the difference in Europe, whose inhabitants could ‘only imagine’ the threat, which was very far away.
We know very well that the uncontrolled attempts of unskilled scholars to repeat our sociological inquiry, without an awareness of the hazards of possible failure, can lead to nonsense. We are prepared to retest all our findings using any other data on the same or similar issue, situation or topic, and we are also prepared to defend our methodological approach against any objections. We are of course also ready to modify, reinterpret or specify our conclusions if there are arguments and evidence strong enough to change our conviction that we controlled our procedures carefully and entered all relevant explanatory factors into our analyses.
I am sure that the Lazarsfeld’s elaboration model is robust enough for the non-standard application we used it for. The methodological approach that we applied in this case was really the only way in which to analyse and interpret our incomplete but extremely interesting and challenging data. I hope that the scientific discussion this paper will hopefully inspire could introduce some new theoretical results into our scientific field and I believe that some of the analytical innovations of our approach could open up a fruitful discussion on methodology.

REFERENCES:
Babbie,E.(1994): The Practice of Social Research. Chapter 16: The Elaboration Model. Wadsworth Publ., New York -, pp. 388-403
Chung, R., Jerabek, H. & Veisová, Eva(2002): 11th September in Cross-cultural Comparissons. Czech Republic & Hong Kong. (Opinions & Communication Behavior in ON-LINE and CATI Surveys). WAPOR 55th ANNUAL CONFERENCE, St. Petersburg Beach, Florida, USA (May 14-16, 2002), CD – ROM Proceedings, Session 5: Public Opinion and Media Responses to September 11.
Hoek, Janet, Gendall, P. & Healey, B.(2001): Web-Based Polling: An Evaluation of Survey Modes. In: WAPOR 54th ANNUAL CONFERENCE, Rome, Italy (September 20-22, 2001), CD – ROM Proceedings, Session A: Internet and survey research, paper A2
Jeřábek, H. & Veisová, Eva (2001): 11th September. International On-line Communication Research. Edition: Sociological papers 01:11. Institute of Sociology Czech Academy of Sciences. (prodej@soc.cas.cz)
Lazarsfeld,P.F. (1955): Interpretation of Statistical Relations as a Research Operation. In: Lazarsfeld,P.F. & Rosenberg,M. (eds.): The Language of Social Research. New York, The Free Press, pp. 115-125
Lazarsfeld,P.F. & Kendall, Patricia L.(1950): Problems of Survey Analysis. In: Continuities in Social Research: Studies in the Scope and Method of ‘The American Soldier’. Glencoe, Ill., The Free Press, pp.133-196
Rosenbaum, P. R., and Rubin, D. B. (1983): The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika, 70 (1): 41-55.
Rosenbaum, P. R., and Rubin, D. B. (1984): Reducing Bias in Observational Studies Using Subclassification On The Propensity Score. Journal of the American Statistical Association, 79 (387): 516-524.
Rosenberg, M.(1964): The Logic of Survey Analysis. New York, Basic Books
Terhanian, G., & Black, G. S. (1999): Understanding the on-line population: Lessons from the Harris poll and the Harris Poll On-line. In the Advertising Research Foundation’s Towards Validation: On-line Research. New York, Advertising Research Foundation.
Terhanian,G., Bremer,J., Smith, Renee & Thomas, R.K. (2001): A Multi-Method Survey Design Approach for Reducing Error in Internet-Based Surveys of Non-Probability Samples. In: WAPOR 54th ANNUAL CONFERENCE, Rome, Italy (September 20-22, 2001), CD – ROM Proceedings, Session A: Internet and survey research, paper A4
Zeisel,H.(1985): Say it with Figures. New York, Harper & Row

Appendix 1
Part of the Original On-line QUESTIONNAIRE – English version
Please Fill-in the Questionnaire!
International Public Opinion JUST IN TIME Research
SEPTEMBER 11th
Thank you for your interest in the questionnaire. It is a part of an
international public opinion research, which aims to reflect the events
that happened in the US on September 11th.
The time needed for filling-up the questionnaire is only 5 minute of your
time. The answers in the questionnaire are anonymous.
Thank you for your help!
Doc. PhDr.Hynek Jerabek
jerabek@fsv.cuni.cz
Charles University, Prague, Czech Republic
Faculty of Social Sciences, Dept. of Sociology
1. Which word would you use to spontaneously describe what happened on
Sept 11, 2001 in the USA?
2. If I ask you directly now and offer a few options, which of them will
you pick as the most/least appropriate? most suitable least suitable
tragedy
assassination
revenge
disaster
war
war attack
attack
terrorism
26. Did you discuss with your friends or relatives the accident?
yes no
27. If yes, how long?
select
The whole rest of the day
more than 1 hour
half an hour
more than 10 minutes
less than 10 minutes
28. Did you discuss these events in more discussion groups?
yes no
29. How you can best express your emotions when you receive the first information?
30. If we ask you directly now and offer a few options, which of them will you pick as the most appropriate?
most apropriate least appropiate
I was surprised
I was astonished
I was shocked
I was afraid
I was angry
I was happy
I was satisfied
I was glad
I was pleased
31. Continent? select North America, South America, Africa, Europe, Asia, Australia and Oceania
32. Country?
35. Religion?  select Christian Moslem Jew Budhist Atheist other
36. Age?
37. Education? select
less than 5 years
5 – 9 years
10 – 12 years
more than 12 years
Thank you, submit!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s