Step 5: Data Analysis

Step five of six in my research process has arrived.  This the data analysis step, or as Norris & Ortega describe it, the scoring research observations step.  But where to begin?

To focus my thinking, I have returned to my research proposal which says:

“The first drafts in the first writing units will be graded according to NCEA criteria to allow placement of students into four groups according to ability which will provide benchmarks for the research. The first drafts in both writing units will then act as pre-tests and the second drafts as post-tests. Situated error, complexity and accuracy measures, which were trialled in a 2013 pilot study, will be used to calculate language development between the two drafts of each unit. A sufficient increase in development should encourage FL teachers to make time in their programmes for students to reflect as they write.”

In practical terms this means I will:

A: Second Language Writing Data

  1. Create an XL spreadsheet with school then student names and the NCEA grade given to their first draft of Task 1 (benchmark).  XL will allow sorting according to school and benchmark as well as intervention and non-intervention groups.
  2. Take all the Task 1 intervention first and second drafts and measure change between them using situated error, then complexity and then accuracy measures.
  3. Take all the Task 1 non-intervention first and second drafts and measure change between them using situated error, then complexity and then accuracy measures.
  4. Take all the Task 2 intervention first and second drafts and measure change between them using situated error, then complexity and then accuracy measures.
  5. Take all the Task 2 non-intervention first and second drafts and measure change between them using situated error, then complexity and then accuracy measures.
  6. Measure the difference between the intervention and the non-intervention participant data for Task 1 in the situated error, then complexity and then accuracy measures.
  7. Measure the difference between the intervention and the non-intervention participant data for Task 2 in the situated error, then complexity and then accuracy measures.
  8. Conclude if there is any significant difference between the intervention and non-intervention groups in both Tasks 1 and 2.
  9. Use Aljaafreh and Lantolf’s Regulatory Scale to measure the depth of the individual students’ responses to the feedback given by the teacher on the first draft.

B: First Language Reflection Data

  1. Prepare students’ digital reflections for analysis by removing the guiding questions and numbering (so only student words remain)
  2. Analyse the students’ reflections with Wordle (or another word frequency tool) to find key words and themes.
  3. If useful, use other quantifying tools to analyse the reflections e.g. co-occurrences (concordancing), distribution of words and/or collocations
  4. Analyse the reflections for depth of thought using Biggs & Collis’ SOLO Taxonomy
  5. Analyse the reflections for level of engagement using Cohen & Dörnyei’s Motivational Instrument
  6. Analyse the reflections using Kember et al’s Levels of Reflection Scheme
  7. Consider Metaphor Analysis

C: Combined Writing and Reflection

  1. Compare progress between the first and second drafts of individual students (case studies) with the quality of their reflections.

D: Return to my “Tools for Locating the Learning,” quadrant and evaluate the tools used in the research.

Quadrant of Tools for Locating the Learning

After my two terms of data collection in four secondary schools, I was concerned I might not have enough data to work with, but I think, now, after breaking things down as above, I have enough to keep me going for some time.

I just have to get started!!!

Teaching and Learning, Horse and Carriage

One of the things I love about teaching is that we are forever learning alongside our students, in fact, reflecting on and providing proof of our learning is a requirement for gaining and renewing practising certificates.

The down side of this, is that when a pleasure becomes a requirement, the magic loses it glow.  However, there is an up side.  With the requirement comes money.  Schools receive a reasonable amount of funding to facilitate teacher professional development and can use it as they see fit.  Like most schools, my school has school-wide initiatives (guest speakers, late start sessions etc) and it facilitates PD initiated by Learning Areas, by Departments and by individuals, which includes contributions to courses, conferences, and tertiary study.

In my opinion, Teaching and Learning, like “Love and Marriage, go together like a Horse and Carriage.”  The teaching with learning union has kept me in this career for more than four decades.  As the old adage goes, it probably took me seven years in the classroom to learn my craft as a teacher and although you can never be bored in the classroom, after a couple of decades, I started to get bored with the career and considered other pathways.  Instead, I went back to university, supported by my school’s PD funds, to learn another language, which I then taught.  Then came the laptop revolution.  I wasn’t convinced that the use of computers improved language learning so I went back to university again to find out.  More recently, I have become interested in the role of metacognition in learning, so have come back to university again to do my own research on the topic.  Again, supported with time and funds from both my school and the wonderful PPTA-MOE initiative of TeachNZ study grants.

So does all my learning make me a better teacher?  It certainly prevents boredom which has got to be good for my students.  Does all my learning aid my students to become better learners?  I think so, but that is very hard to quantify.  However, I feel I have some indications.  Last year, I used my own students for a pilot study on reflective processes and, for only the second time in eight years at my school, there was a 100% pass rate in NCEA in those two classes.  Although not pure evidence, it is an indication that teacher professional development improves student learning.

Te wiki o te reo Maori

Last week a young man in Year 11 at Kapiti College became a hit on Youtube with his NCEA assessed English speech on cherishing the Maori Language by making an effort to pronounce it properly.  He stated that the Maori language is part of our heritage as New Zealanders.

As I watched the Youtube video, I realised that the popularity of his speech represented a cultural shift that is worth celebrating in te wiki o te reo Maori, Maori language week.

I enjoyed a childhood enriched by Maoritanga.  In Christchurch in the 1950s, I spent much time with an aunt and cousins who were Maori. We were close.  In the early 1960s, when my parents moved north to Tauranga for work, I played at the pa with my neighbours, learning to track in the bush with them and to sing and speak the snippets of Maori language that punctuated their everyday greetings, games and music. My standard three teacher was a young Maori woman who taught me to do the poi after school. Then my parents moved me to Auckland for secondary school where I belonged to the Maori Club and performed in assemblies and concerts. I thought my inclusion was normal and took it for granted.

But then came the radicalism of the 70s and the anger of 80s, most of it justified.  But during those years, I experienced being not welcome to join Maori interests by radicals who saw all pakeha as foreigners who had a choice to go back to where they had come from (which I couldn’t)  I was shocked and hurt because my identity as a New Zealander and my connection to Maori and the Maori language were being challenged.

Now several decades later, after having taught languages since the mid-70s, including for two years Maori language,  I hear this speech but more importantly witness this national applause. There has indeed been a shift if a young pakeha male can be celebrated for standing up for the Maori language without being challenged or questioned as to his right to do so by Maori. I am encouraged.

Kapai, e hoa!

Really Simple? How do links and feeds differ?

I have known about RSS feeds since 2004.  I remember exactly when because of the job I had just started that introduced them to me in the blog context.

The way I tend to get personal feeds is by receiving them directly into my personal email box.  At work I have opted to create tabs on my browsers’ home pages and use the favourites facility.  In one browser, the tabs link to the sites I use daily.  In a second browser, I have created favourites tabs according to the classes I teach.  Most of the links in those folders are used annually as I teach certain topics.  Are these two methods the same as RSS feeds?  I am not so sure.  The one is push technology and the other is pull.

In addition, the ways I stay connected at work have their issues, if I need to update my computer which I have done twice since 2004, I have to create those folders again, and even though I have tried to save my favourites and transfer them to the new computer, I have never achieved that successfully.  Consequently I have had to start all over again finding those precious web links.  Having an online site like Feedly would ensure I don’t loose those links/feeds in the transfer of data.

So thanks for the info about online sites for RSS feeds, 23Research!  I think I will have a go with Feedly.

Writing to Clarify Thinking

I have discovered over the years that in order to clarify my thoughts and come up with new ideas and plans I need to talk or write “out loud.”

For the past six months I have been collecting data from four co-educational secondary school, second language classrooms across the city.  After the intensive two year process of reading and writing that led to doctorate approval in November last year, this past six months have been comparatively low key, in fact, I confessed to my supervisor that I feel I have ‘gone off the boil.’  I haven’t been reading or connecting with my academic community in the way I needed to up to this point.  Martin assured me that this was normal and that as I began analysing and writing up my data, I would find direction for my reading again.

So now that my simmering data collection period is over, I need to apply some heat and get boiling again with some data analysis.  But where to begin?  What advice would I give my own students?

Here’s what I am thinking:

Step One:  Set aside a sacrosanct time and place in which this work will occur.

  • I have already negotiated with my school, through my TeachNZ study support grant, so they have given me every Tuesday this term where a relief teacher will take my classes and I can stay at home or go into the university to analyse my data.

Step Two: Organise my raw data so I know what I have and don’t have.

  • These school holidays, after spending a week marking class work, I have had time to save and sort my data into googledrive folders and make certain I am the owner.  (I have learned from bitter experience that this is an essential step if I don’t want participants to take the data away with them at the end of the school year (when students leave schools, their school google accounts are discontinued by IT staff and access to data ia lost unless you ‘own’ it.)
  • I have also gone back to the participant teachers to fill some gaps found in the data set.

Step Three: Return to my approved proposal, in particular the methodology section, and recall what I said I would do.  I wrote:

  1. “In order for the research in the proposed context to be robust, it will follow recommendations by Norris and Ortega (2005). They suggest beginning by defining the constructs (psychological qualities) to be measured, identifying the behaviours to be observed, and specifying the tasks that will elicit such behaviours…”  I fufilled these steps when I wrote my proposal and gained doctoral and then ethics approval but I need to remember what they are and write them out again.  Maybe a table?
  2. “…They then suggest controlling or accounting for variables when gathering the data,…”  This was pertinent last term when gathering data.  I will need to report information from the field notes I wrote in my final thesis document.
  3. “…scoring the observations in a way that connects them to interpretations, and finally summarising and analysing the data according to probable categories…”  These are the final two stages in the research and the work I need to embark upon..

Just even realising that I have completed the first four of Norris and Ortega’s (2005) six stages is encouraging.  However, now we get to the nub of the matter, scoring the observations in a way that connects them to interpretations.

Start date:  Tuesday 20th July, 9 am, at my desk, paper and digital folders at the ready.

Norris, J., & Ortega, L. (2005).

Bellowing to the Void

One day last summer, I heard some appalling bellowing coming from the direction of our front driveway.  I discovered it was the advertising call of a neighbour’s ten year old daughter who was trying to sell flowers she had picked from her garden to the surrounding residents in order to earn money to spend at the community fair the following day.  The yelling went on for an hour before I finally lost patience and asked her to move on.  In that time nobody had come to buy but many had been irked.

Blogging for Research feels a little like this.  I am currently taking part in a programme called 23 Things for Research. Having set up one’s blog and thought about security, Thing Four asks participants to explore the blogs of others.  Assumedly, it is designed to encourage cross-semination in blogging through the reading and commenting on other people’s blogs. Good concept, but how does it gain momentum?  How do you acquire an audience for your blog and thereby some genuine interaction with those who can contribute to your academic thinking?

So far the stats indicate that in the past eight days there have been about 15 visitor views to my blog site (most likely many from the same visitors) and three of those visitors have left comments, for which I am grateful and encouraged.  I suspect, however, they are administrators or teachers in the 23 Things for Research programme.

So how does one become ‘read’ by a genuine contributing audience who will enter into useful academic dialogue with the writer around the research topic?  How does one avoid standing at the gate and bellowing to the void?

Maybe instead of musing about blogging, I shall start reflecting on reflection, the topic of my research, and see what transpires.

The Etiquette of Academic Blogging

I recently read a story in the New Zealand Herald about a Formula 1 star who had been invited to lunch with the queen.  He had tried to strike up conversation with her during the first course.  Wrong move.  She politely informed him that during the first course she spoke with the person to her right and he should converse with the person to his left.  During the second course she would speak to him and so on around the table.

In my last post, I quoted myself from a response I gave to another blog. I basically cut and pasted most of my response into my own blog and linked to original.  It got me thinking…

Is there an etiquette to blogging that new bloggers would do well to discover earlier rather than later?  I would appreciate being told by a blogger-king or -queen, if this is the case.

Teacher or Student?

I have just commented on Derek Wenmoth’s blog regarding what makes effective teacher PLD.  For those who are not in education that means Professional Learning and Development.

As an EdD student, I am also a teacher in her 42nd year of teaching.  I do feel odd in my dual role from time to time.  It feels most odd when I am treated as an inexperienced student by university teachers.  Good for the humility, for sure.

Back to Derek Wenmoth’s blog again… You would think I might know a thing or two about teacher PLD, having experienced all sorts over the past four decades.  I know what works, in other words, what helps me teach most effectively and what helps me raise achievement levels for my own students in my own school.

As I suggested to Derek, confirming his post about the recent publication, Teachers Know Best, that “in order to be effective, teacher PLD needs to be teacher-initiated.

I love the phrase that I think comes from Dr Lorraine Munroe, that secondary schools have “over-permeable boundaries.”  We are pushed and pulled from all directions – students, parents, school leadership, community, subject associations and advisors, national curriculum and assessment requirements, and more often than not their demands are contradictory.

Teachers are professionals who have spent many years training in the university and many more years training in the classroom, and who actually know a thing or two about their field of expertise, including, what will aid the teaching and learning that occurs in their own classrooms.

Effective PLD has to be teacher-initiated or else it becomes an exercise in compliance and box ticking.”

The best PLD for me and my students is my being an EdD student and researching matters that are relevant in my classroom amd therefore possibly but not necessarily in the classrooms of my colleagues.

How do I thank them?

Despite the data collection discrepancies experienced in the reality of the four classroom contexts of my research, I have to admire the good will and graciousness of my four colleagues, who without hesitation offered their students to be part of my research, and who didn’t flinch as my research demands grew like Topsy.

In an NCEA context, we are so tight for teaching time, and yet they, all four, were willing to offer me precious teaching time with their classes.  I can’t thank them enough.  In fact, I owe them.

I once took part in some research and was offered a $15 voucher which I considered an insult.  Nothing would have been better.  I don’t want these professionals who I respect to feel the same way.  Maybe a $50 voucher?  Some sort of gift?  Any ideas, anyone?

The reality of data collection

Collecting data in school contexts is messy.  In fact I would go so far as to say that the context makes a mockery of the finickity ethics approval process.

I had set up a neat counter-balanced research design using four co-educational secondary school second language classrooms.  It ended up being anything but neat.

In the two research rounds, all sorts of unplanned incidents occurred.  We had students forgetting to go to the computer room, arriving late and having no time for their reflections, the actual research in other words – they did it for homework; we had students absent and late and thereby not getting the same instructions as others; we had students deleting their reflections before I had transferred ownership to myself; on one occasion the birthday celebration of an exchange student, complete with cake, took the first fifteen minutes of the research period; on another occasion, when everything was going well, the students’ flow of reflection writing was cut short by ten minutes as the teacher prepared the students for the bell.  Then there was the slow return of PI forms. We have ploughed on regardless.  Hopefully they all come in in the end.  So now I am ready to see what I can make of it all.

How do you avoid birthday cake and unintentional sabotage when there is no second chance?  I could have explained the research better to the teachers concerned.  Never-the-less, I am so grateful to them for having taken part.  It was actually me that interrupted their programmes and stole their precious time, not the other way round.