Reading Reflection on “The 4th Paradigm of Science”

First of all, I’m really saddened by the missing of Jim Gray at sea. I’d like to express my deep condolences to him, and the recently missing MH370 plane, and all other planes and boats that have gone missing in the ocean.

According to this reading, the 4 paradigms of science are experimental, theoretical, simulation (computational) and data-intensive scientific discovery. But in my experience and observation, for most people today — researchers or not, their understanding of rigorous scientific research is still experimental.

For experimental research, the researchers usually have a question and a hypothesis about this question, then they would go out and collect some data to test this hypothesis. The scale of the data collected is usually less than thousands. These data are all collected for the purpose of answering the specific question, the data collection process is carefully designed to reduce bias, and increase generalizability and representativeness, so they are usually high quality data.

Nowadays, digital devices are everywhere, and they record volumes and volumes of data. Large part of the data were not recorded for answering specific questions. So “what do we do with these data? how do we turn them into insights?” are questions for the 4th paradigm of scientific discovery. Therefore, many research in this area are exploratory. Researchers in this area often get questioned by experimentalists “so what is your hypothesis?”

It goes without saying, the datasets for data-intensive science is large. However, large doesn’t mean good. Because these data are not collected to answer your specific research questions, so it takes great effort to curate the data in order to make them useful. As this reading says, there are 3 basic steps for data-intensive science: capture, curate, and analyze/visualize. Data curation could be the most time-consuming part. Therefore, Jim Gray advocates funding for generic data curation and analysis tools. However, my questions are (or parts that I don’t understand), is there really a generic way of cleaning and curating data? Aren’t any method of data cleaning essentially a type of subjective bias? Are there successful Laboratory Information Management Systems (LIMS) today 7 years later after this concept was brought up?

Tony Hey, Stewart Tansley, & Kristin Tolle. (Eds.). (2009). The 4th Paradigm: Data-Intensive Scientific Discovery.


Reading Reflection: Where does the “Natural” Come from in Natural Design?

This week’s CGT512 reading chapter1 of “The Design of Everyday Things” by Donald Norman emphasizes natural design. For example, doors have to be designed with proper visual clues so that people will naturally know which side to open and close the door, as well as to pull or push the door. I understand that for relative simple designs as doors, there maybe a common consensus regarding which way is natural for most people, but for complicated computer systems, what is natural may mean different to different people. After all, what is “natural” and where “natural” comes from? Are the feelings of natural come from previous experiences and previous training? For example, I’ve used windows OS for years before I switched to Mac, then I would feel many things in the Mac OS odd at the very beginning. I still think iTunes is oddly difficult to use until now. Some people would feel the so-called “natural scrolling” on the new Mac OS Lion unnatural, because they’ve used the opposite way for so long. The relatively elderly people would feel everything electronic unnatural for them, but the millennial generation was born with them and feel all these electronic systems natural. For doors, because it has been existing for so long, so everybody knows about doors, but for electronic systems which did not exist at human origin, whether they are natural or not, depend at least partially on the generations or groups of people who live with them or were born with them. The feelings of natural at least partially come from people’s environment or habitat. With all these being said, I still believe there are certain level of common consensus of natural feelings wired into human brains that can be integrated into complicated computer systems design, such as movement, gestures, touch, orientation of directions, color, etc.

Another question kept popping up in my head is that what is the relative importance of rigorous UX research and intuitions of talented people? The UX book by Hartson and Pyla laid an iterative UX process lifecycle, but what I got from reading Steve Jobs’ biography (correct me if I’m wrong) is that Jobs never did user study, he just did what he believed was good for the users, and he hired very talented people to do the design. Many aspects of many Apple products get cited as good design practices in many UX textbooks, but they seem not necessarily produced based on the UX process lifecycle, they were just produced based on the understanding of a group of highly talented people. These geniuses tell the users what they want, and then the products they designed were put into UX textbooks to train future UX researchers. Are genius intuitions better than rigorous research results?

Qualitative Spirit

I am deeply confused by many things this semester, and I felt I am almost near a breaking point at some moments. One reason is that I felt I am oppressed by some beliefs about conducting scientific research that are contradicting with part of my value system, but I don’t know how to defend myself for various intellectual and practical reasons. I don’t know whether I should defend myself, or what to defend. I don’t know whether I should learn to make compromise or is there a real compromise to make. There are, of course, various other social, emotional, practical and intellectual reasons resulting my situation now. Maybe all of these are normal during the process of growing up. I am going through some kind of transition and I don’t know how long it will take. All I can do is to be patient, and do whatever I should do. Worries and concerns will not do me any good at this moment.

I am learning qualitative research methods this semester, and the following are some excerpts from qualitative methods books. They resonate with my belief about research and sort of make me feel better.


“No one should plan or finance an entire study in advance with the expectation of relying chiefly upon interviews for data unless the interviewers have enough relevant background to be sure that they can make sense out of interview conversations or unless there is a reasonable hope of being able to hang around of in some way observe so as to learn what it is meaningful and significant to ask.” (Dexter, 1970, p. 17).

“Well-structured, focused questions are generally the result of an interactive design process, rather than being the starting point for developing a design.” (Maxwell, 2005, p. 66)

I got to know that changes happen during research, and it is normal. There are practical reasons such as IRB review, financial and time limits that constrain changes during research, but I should not feel bad when changes happen. They do happen!

Personal Goals, Practical Goals, and Intellectual (scholarly) Goals

“Traditionally, students ave been told to base this decision [of the topic, issue, or question selected for study] on either faculty advice or the literature on their topic. However, personal goals and experiences play an important role in many research studies.” (Maxwell, 2005, p. 16)

“Choosing a research problem through the professional or personal experience route may seem more hazardous than through the suggested [by faculty] or literature routes. This is not necessarily true. The touchstone of your own experience may be more valuable an indicator for you of a potentially successful research endeavor.” (Strauss and Corbin, 1990, p. 35-36)

“Traditionally, discussions of personal goals in research methods texts have accepted, implicitly or explicitly, the ideal of the objective, disinterested scientist, and have emphasized that the choice of research approaches and methods should be determined by the research questions that you want to answer. However it is clear from autobiographies of scientist (e.g., Heinrich, 1984) that decisions about research methods are often far more personal than this, and the importance of subjective motives and goals in science is supported by a great deal of historical , sociological and philosophical work.”  (Maxwell, 2005, p. 18)

“In addition to your personal goals, […] there are practical goals (including administrative or policy goals) and intellectual goals. Practical goals are focused on accomplishing something–meeting some need, changing some situation, or achieving some objective. Intellectual goals, in contrast, are focused on understanding something–gaining insight into what is going on and why this is happening, or answering some question that previous research has not adequately addressed.”   (Maxwell, 2005, p. 21)

I got to know that research questions are usually based on intelectual goals rather than practical goals, and actually, questions based on practical goals are usually not directly answerable. Practical goals are the “so what” piece, the implication of the research, and are of particular importance for justifying the research.

Reflection on the Dark Side Presentation

First of all, I think we are very brave to try out one class remotely. It is very important experience for a class studying the Internet. I don’t know whether previous students in this class have tried remote classes or not, I hope Dr.V could let students in the future continue to try, although it may not be a pleasant experience. Hope students in the future could find a better software other than Adobe Connect to do this. Any revolution or progress will not happen if we always shy away from unpleasant possibilities. The key here is not to avoid the wrong things, but to know what is wrong, and make the wrong things right.

Second, regarding what is wrong, I think the key failure point is that each of us has too much freedom for ourselves, but not enough freedom for participating the class with Adobe Connect. We cannot see others if they are not the current presenter, and other people cannot see us if we are not presenting. Other people cannot hear us either if we do not press the “talk” button. We lost the supervision, and self-discipline doesn’t work all the time. Basically we could do anything else–eat, do homework for other subjects, check Facebook feed, etc. We had too much freedom, and I ate too many milk duds while listening to others’ talk. Adobe Connect may be perfect for people who have to attend boring conferences all the time, but definitely not for engaging students in class.  Although all of us had talk priorities, we had to press the “talk” button at the risk of interfering the presenter, overloading the network and introducing lots of echos. So many of us would rather type in the chat box rather than speak. The social affordance of Adobe Connect doesn’t support free discussion as much as in Google+ Hangout.

Finally, Adobe Connect is designed for formal conferences, and Google+ Hangout is designed for informal friendly hangout. Experiences of using them for Tech621 made me realize they do have lots of differences depending on their specific purposes. Any such software in the future, if the purpose is for remote class, should consider how to engage students (let students want to and feel easy to participate) as the highest priority.

RAA#3: CommentSpace, Collaborative Visual Analytics

  1. APA Citation:
    Willett, W., Heer, J., Hellerstein, J., & Agrawala, M. (2011). CommentSpace: structured support for collaborative visual analysis. Proceedings of the 2011 annual conference on Human factors in computing systems (pp. 3131–3140). PDF

    CommentSpace website

  2. Purpose:  (1) Present details of a web-based collaborative visual analysis system CommentSpace that aims to help users better make sense of the visualizations through synthesizing others’ comments. CommentSpace “enables analysts to annotate visualizations and apply two additional kinds of structure: 1) tags that consist of descriptive text attached to comments or views; and 2) links that denote relationships between two comments or between a comment and a specific visualization state or view. The resulting structure can help analysts navigate, organize, and synthesize the comments, and move beyond exploration to more complex analytical tasks. (2) Evaluate this system: “how a small, fixed vocabulary of tags (question, hypothesis, to-do) and links (evidence-for, evidence-against) can help analysts collect and organize new evidence, identify important findings made by others, and synthesize their findings” and “establish common ground”.
  3. Methods: (1) present technical details of the design of this system, and usage scenario (2) evaluate by two controlled user studies and a live deployment comparing CommentSpace with a similar system that doesn’t support tags and links.
  4. Main findings: (1) A small, fixed vocabulary of tags and links helps analysts more consistently and accurately classify evidence and establish comment ground. (2) Managing and incentivizing participation is important for analysts to progress from exploratory analysis to deeper analytical tasks. (3) Tags and links can help teams complete evidence gathering and synthesis tasks and that organizing comments using tags and links improves analytics results.
  5. Analysis: (1) This paper is from the “garden” of information visualization and visual analytics. This line of work (collaborative visual analytics) is drawn from and expanding into CSCW and social media research. Because computing systems are eventually serving people within their social contexts, also because of the popularity of the web, many technical systems are implemented on the web and thus seek to support people, their communication and collaboration. I see this emerging converging point between social media and visualization techniques, but there are still huge discrepancies in the way of thinking and doing among researchers in different disciplines (esp. computer science and social science). Traditionally, the way of conducting user studies in technical world usually lack of rigor or depth. “It was almost a joke in some technical domains that reviewers of papers just need to check the mental box of the existence of user studies without considering the quality”. Large part of those papers are dedicated to “fancy algorithms”. The future of social computing calls for close collaboration between computer scientists and social scientists, further more engineers, artists and designers. (2) This paper is related to my project of integrating user participation in rating, tagging and commenting academia papers. CommentSpace is designed as a modular softare that can run  in conjunction with any interactive visualization system or website that treats each view of the data as a discrete state, so maybe I am looking forward to adopt it or some elements of it to my project in the future.

Reading reflection: Attention

I will mention why I put this video here at the end of this post.

I briefly summarize a few papers and add some of my thoughts following each two papers.

01. Is Google Making Us Stupid? What the Internet is doing to our Brain,  by Nicholas Carr

Claim (argument): The Internet is changing our way of processing information to become shallow and make us less capable for concentration and contemplation.

Evidence: Start with personal experience of not being able to concentrate on reading and writing–>Expand to friends, acquaintances (literacy types)–>Bloggers he followed–>Scholarly studies that the web is changing the way we read and think ( simple, clear and well-structured way of generalizing and developing the argument)

02. Distracted: The Erosion of Attention and the Coming Dark Age, by Maggie Jackson

Claim (argument): Modern technologies is eroding our attention, thus the ability of deep, sustained focus and analytical reasoning as a society. We are facing a real risk of social decline if we do not nurture our attention.

Evidence: parallel to history; ADD; busyness, multitasking; empirical studies about kids multitasking, low patience, and lack of analytical reasoning ability; Power Point; Theories about attention and it’s importance, etc.

My thought on the topic of the two papers above:

I think maybe we can make an analogy of the society growing to a human child growing. My mum always tells me how she was astonished about and even “admirable” to me about how much I could concentrate on things when I was around 1-3 years old. When I started to learn how to use  scissors, one night, I was using the scissors to cut papers and I refused to go to bed, when my mum woke up from her sleep, I was still cutting papers maybe already for 2-3 hours. I was so concentrated and forget about the time, or I think I didn’t really have the concept of time, all things in my world was scissors and papers.  Another time, I would sit on the floor and play lego (the Chinese version of similar building block toy as lego) for 2 hours nonstop.  However, as I grow up, I start to be aware of time, I start to need to process more and more information, and my world is not that pure anymore. I feel I could never go back to that kind of mental status anymore even when I try my best to concentrate. When I spend too much time tackling some hard problems I get panic about time, because I have so many other things to do, so I start to look for some easier solutions, then I lose the spirit of never giving up, drilling on the hard part, and deeply polishing my analytical reasoning ability. This is almost inevitable as a part of growing up. Most adults’ worlds are much more complicated than children’s. We have to consciously process lots of information, and we need to worry about lots of things. The growing process is also the process that we go out of the ego stage (Piaget developmental theories) and start to be aware of the outside world, the cost is that we are inevitable exposed to more information, and maybe distracted by this information.

As a society, we are growing like a human child. As we growing, we inevitably need more information and communication with others. Modern technologies esp. the web went a long way to promote this growing by allowing easy information diffusion and communication among people, so we become aware of way more things than before. Also, inevitably, we get distracted by these things, but this is part of growing up. We are getting more mature, we need to learn how to deal with this and cultivate our lost attention.

If this analogy is appropriate, is the society going to die some day just like a human being is going to die eventually? Do modern technologies somehow speed the dying of the society at the same time it speeds the growing of the society? How old is the society now? Is death the dark age Jackson predicted?

05. Self-interruption on the Computer, by Jing Jin and Laura Dabbish

Research goal: Why do people interrupt themselves on the computer and switch to doing something else? What do internally generated interruptions look like in practice? What are their potential negative and positive side-effects?

Methods: Shadowing observations of 13 participants doing their normal work tasks on computers followed by 30min to 1 hour retrospective interviews; grounded theory; theoretical sampling to only analyze the internal interruption data; verified using independent coders

Main findings: A typology of self-interruption on computer. Seven types of self-interruptions were identified: adjustment, break, inquiry, recollection, routine, trigger, and wait.

06. Multitasking and Monotasking, by Dario Salvucci and Peter Bogunovich

Research goal: To test a claim that when users are alerted to interruptions at points of higher mental workload, they delay processing of the interruption until they have reached a point of lower metal workload.

Methods: 20 users participated the study; Participants have to do a mail task on customer support, while answering interrupting chat messages. The chat prompt was generated at a pseudo-random point: the system tracked the user’s events, after one of eight different events, triggered a chat prompt 50-200ms after the event. Users can choose to defer the chat. System recorded all the transitions and researchers analyze the transitions between events.

Main findings: Verified the claim. The experiment also helps to clarify one source of mental workload, namely the problem state—temporary information needed for task processing.

My thought on the topic of the two papers above:

The second paper above mainly talks about external interruptions, but it also says in the discussion section that “we would suspect (though further research would need to be confirm) that this ability also generalizes to user self-interruptions and discretionary multitasking”. I highly doubt this assumption/hypothesis. Users maybe capable to deal with external interruptions when they are inconsistent with the users working mode, however, internal interruptions are more difficult to control.

07. The Laptop and the Lecture: the Effects of Multitasking in Learning Environments, by Helene Hembrooke and Geri Gay

Research goal: To test two hypothese:

H1: Students in the open laptop condition would perform significantly poorer on immediate measures of memory for the lecture material.

H2: The memory decrement observed would not result from the relatedness of the content viewed in the secondary task (laptop use) to the primary task (lecture information). In other words, content relevance would not contribute significantly to the variance observed in the main effect above.

Methods: Two groups of students (22 in each group). One group was encouraged to use laptop during the lecture, while the other group was not allowed to use laptop. All the students were tested about the content of the lecture immediately after the lecture. ANOVA analysis.   

Main findings: The two hypotheses were proved, but this is only for immediate test of memory after the lecture. For the overall performance, the average score of using laptops in class is B+, which is very good. This is largely because the structure of the class was nontraditional, highly interactive and dynamic. Had the class been more traditional and grades determined by conventional test of memory the outcome of students who use laptops may have been different.

09. Caught in the Web: university student use of Web resources, by Yu-Mei Wang and Marge Artero

Research goal: 1. How do students use web resources (academic vs. non-academic); 2. What are the students online search behaviors? 3. What are the students’ perceptions of web resources? 4. Do students evaluate information on the web? 5. What types of training do students perceive they need to successfully utilize web resources?

Methods:700 questionnaires about information literacy skills. SPSS.

Main findings: the findings of this study show that there is an urgent need for students to develop information literacy skills and apply these skills in the electronic information environment.

My thought on the topic of the two papers above:

Whether or not to use internet in classroom has been a huge debt. Some think technology will fail education, as it can hugely distract students. Most say technology will save education. The question needs to be discussed is not whether or not to use, but how to use. How to guide the students to properly use, and how to change the structure of the class and the assessment methods to be more supportive for using web in class. There is a digital ethnography video provoking lots of thoughts on this. I loved this video a lot. I put this video at the beginning of this post to call for your “attention”.