Open Access

Original Article, Biomed Biopharm Res., 2022; 19(2):397-409
doi: 10.19277/bbr.19.2.298; PDF version here [+] ; Portuguese html version [PT]  

Supplemental Material - Appendices A-D [+] pdf here - DOI: 10.19277/bbr.19.2.298_Supplemental.pdf

 

Pretest questionnaire for anti-ageing cosmetic claims substantiation: a description of validation of efficacy and sensory perception questionnaires

Mariane Massufero Vergilio 1, Laura Moretti Aiello 2, Tamiris Anselmo 2, Gislaine Ricci Leonardi 1,2*

1Graduate Program in Internal Medicine, School of Medical Sciences - University of Campinas (UNICAMP), 126, Tessália Vieira de Camargo St., "Cidade Universitária Zeferino Vaz," 13083-887 Campinas, SP, Brazil; 2School of Pharmaceutical Sciences - University of Campinas (UNICAMP), 200, Cândido Portinari St., "Cidade Universitária Zeferino Vaz," 13083-871 Campinas, SP, Brazil

* corresponding author: This email address is being protected from spambots. You need JavaScript enabled to view it.

 

Abstract

The application of questionnaires for the assessment and collection of data on target audiences and new products is a very common practice in the field of cosmetics. However, to obtain reliable results, it is essential to pretest the instrument to detect possible problems and inconsistencies. The aim of this study was to demonstrate the entire pre-testing process of two questionnaires used in the cosmetic claims substantiation area; a skin self-assessment questionnaire and a sensory analysis questionnaire for cosmetic formulations. According to the pretest protocol analysis, 15 women were interviewed for each questionnaire. All difficulties were recorded, categorized, and analyzed. Subsequently, the inconsistencies found were corrected in the final version of the instrument. During the validation of the questionnaires, the biggest problem faced was related to the application of scales and technical concepts within the questions. This difficulty means that respondents are unable to represent their real perception of the evaluated product. This problem can be solved by adding detailed explanations in the footer or within the wording of each question. In conclusion, learning and applying the pre-test in questionnaires can ensure greater reliability and clarity in research results in the cosmetic area.

 

Keywords: Questionnaires, cosmetics, substantiation of claims, self-efficacy, sensory attributes

Received: 25/11/2022; Accepted: 30/12/2022

 

Introduction

Consumption and demand for anti-ageing cosmetic and dermatological strategies have grown exponentially in recent years, due to increased life expectancy and a new culture focused on health care and aesthetics, bringing an improvement in the quality of the population (1–3). Consumers are increasingly concerned about the components of the products they are using, considering quality, effectiveness, safety and sustainability (3).

In this context, cosmetic claims related to aging gain prominence. Different methodologies can be used to provide information to help evaluate the effectiveness of cosmetic products to reverse or prevent the signs of skin aging (4). These include scientific tests based on the consumer's observation of the effectiveness and function of the product, as well as tests in which participants evaluate the sensory properties of a given product (4).

Of the number of available products, consumers give great preference for those which are pleasant to the senses, an extremely important attribute in the choice of the product in view of the numerous existing options that have the same function (5). Allied to sensory parameters, the study of the effectiveness of products, through the perceptions of consumers, is essential to prove their purposes, in addition to providing information for the development and marketing of new cosmetic and dermatological formulations, reformulation of existing products and optimization of the formulation process (6,7).

For these evaluations to present reliable results, it is necessary to use questionnaires elaborated from a validated measurement scale and appropriate to the study (8). Therefore, the development of these instruments is a complex process consisting of several steps toward achieving the proposed objective (9).

Thus, for the structuring of the questionnaire, it is necessary to first specify the information such as the central problem of the instrument, the necessary approach, the issues related to the theme and, principally, the target audience, which exerts great influence on its development (9).

To ensure the correct comprehension of all questions in the questionnaire and to identify possible difficulties, the pretest is an essential step during the development of the instrument. Through this, problems such unfamiliar words to the interviewee, ambiguities, and inappropriate answers to the aim of the research can be evaluated, enabling the adequacy and modification of the preliminary or initial questionnaire. In addition, the pretest also allows verification of structural aspects such as the number of questions, how they are performed, and the order in which they are arranged (10).

To encode the obtained data in the pretest stage, it is very important to analyze the content that consists of a methodology for reading and interpreting the interviewee’s answers, which also does not escape the personal interpretation of the interviewer (11,12). This analysis consists of three phases: the preanalysis, categorization, and treatment of the results (13).

The aim of this study was to describe the pre-testing of skin self-assessment questionnaires and another of sensory analysis, both focused on substantiating anti-ageing cosmetic claims, in order to indicate problems and solutions encountered during their development, to ensure greater reliability and clarity of data collection instruments in studies in the cosmetic area.

 

Material And Methods

Experimental design

Two initial questionnaires were evaluated, one for the self-evaluation of skin appearance (Supplemental Materials Appendix A) and another for the sensory evaluation of cosmetics (Supplemental Materials Appendix B).

The pretest of each questionnaire was performed through an interview of a sample, nonprobabilistic and for convenience, of 15 participants per questionnaire (n=30) (9). All participants signed an informed consent form. Women between 30 and 60 years old, consumers of anti-ageing cosmetics, were selected.

The protocol analysis process was chosen for the performance of this initial test, in which the interviewee “thinks aloud” when answering the questionnaire, and commonly, the interviewees’ observations were recorded by audio, and analyzed to determine the reactions evoked by different parts of the questionnaire (9). The entire interview was recorded on a cell phone for further analysis. The interviews were partially transcribed, focusing on the moments when the interviewees showed difficulty or some change of mood.

One experienced and one non-experienced interviewer were used. Experienced interviewers quickly detect insecurity, confusion, and resistance from respondents when answering questions. The non-expert identifies problems related to the interviewer, ensuring that the test application is robust, since the result of the research needs to be independent of who is applying the test (9).

The study was approved by the Research Ethics Committee of the School of Medical Sciences of UNICAMP (CAAE: 13367219.5.0000.5404).

Data collection instrument

Self-evaluation of skin appearance questionnaire

The initial structure of the questionnaire can be found in Supplemental Materials Appendix A. This questionnaire was developed with the aim of providing data regarding the appearance of the skin. The parameters evaluated by this questionnaire were hydration, firmness, elasticity, flaccidity, wrinkles and expression lines in the lips/eyes/forehead, luminosity/freshness, signs of tiredness, skin tone uniformity, intensity and number of spots, vitality, apparent pores, presence of dark circles, bags beneath the eyes and lifting effect. Questions were used to collect structured data, using different monodic interval scales to measure attributes, according to the standards of substance of cosmetic claims (14).

Sensory evaluation of cosmetics questionnaire

The initial structure of the questionnaire is found in Supplemental Materials Appendix B. This instrument addresses the sensory profile of a tested cosmetic topical formulation. To obtain the sensory profile of the formulation, the sensory attributes that refer to the product itself were selected during the application process: consistency/texture, perfume/fragrance, absorption time and spreadability. Skin attributes were also selected to be evaluated after the application: stickiness, oiliness and softness (6,15–17).

Data analysis

The answers obtained with the pretest were coded and analyzed (9). For the analysis of the qualitative material obtained in the interviews, the simple method of thematic content analysis was used (18). In relation to the quantitative data analysis process, the following steps were followed: establishment of categories; coding and tabulation; statistical analysis of the data (frequency, arithmetic mean, percentages, standard deviation, with aid of sector charts and others) (10,18).

After identifying and analyzing the problems encountered during the pretest, the questionnaires were edited and corrected (9).

 

Results

From the qualitative evaluation of the content of the interviews, the difficulties and suggestions pointed out by the participants during the interview were categorized according to Table 1.

The sociodemographic characteristics of the 15 women who pretested each questionnaire (sensory analysis and self-assessment) are described in Table 2.

Self-evaluation of skin appearance questionnaire

The frequencies at which misunderstandings and difficulties occurred in each category during the application of the pretest of questionnaire 1 are shown in Table 3.

As indicated, regarding the conceptual difficulties of questionnaire (Supplemental Materials Appendix A), it is verified that, in the item Medical and General History, the participants presented doubt in the concept of “neoplasia” and in the concept “psoriasis”. For this reason, a simple explanation of the health conditions was added to the question itself: “Neoplasms (tumor)” and “Psoriasis (chronic autoimmune skin disease, characterized by reddish spots covered by whitish scales)”.

Regarding the parameters evaluated by the intensity scale, there was some difficulty in understanding the words “lushness” and “luminosity” (Question 20), since the participants erroneously relate the terms to skin oiliness and sweat. Thus, he chose to remove this question from the questionnaire. In addition, some interviewees reported not knowing the definition of “elasticity” or cold not distinguishing it from the meanings of “firmness” and “flaccidity”. Therefore, we removed the question regarding the flaccid attribute, as it is the most confusing term, according to the respondents, and added the following descriptions to the respective questions: “Firmness is characterized by firm skin, with tonus, consistency, support, opposite of flaccidity” and “Elasticity is characterized by flexibility of the skin, undergoes deformation when subjected to traction, and partially or totally returns to its original shape”. Some participants did not understand the attribute “signs of tiredness” and “skin uniformity”. The questions were considered vague and confusing by the interviewees. In these cases, phrases explaining the meaning of each item were also added.

In the data collection instrument, conditional difficulties were also identified in six questions. In question 10, the elements “aesthetic procedures” and “rejuvenating cosmetics” are evaluated in the same question. However, since only the item “aesthetic procedures” is exemplified in the question, it is not clear whether the use of cosmetics should be described with reference to performing these procedures together or not. To avoid this difficulty, the two elements were evaluated and exemplified in different questions in the modified version of the questionnaire.

The non-specification of a reference period in questions 29 and 30 made it difficult to evaluate the attributes by the volunteers who questioned whether the answers should be given based on the moment the instrument was applied or whether previous times should also be considered. The participants reported that the presence of bags under the eyes and dark circles depends a lot on the time of day and on factors such as sleep and stress. For this reason, time indicators were placed in the questions, such as, “Regarding the bags under the eyes, would you say that in the morning (...)”.

The difficulty of interpretation on the Likert scale was also reported by the participants, especially during the self-assessment of clinical attributes by the numerical scale of 7 and 9 points. In questions 14 and 15, some interviewees questioned the difference in the levels present between these frequency scales. To overcome the difficulties presented, the numerical scales were complemented with the corresponding explanations and numbers at each level.

All changes made in the questionnaire (Supplemental Materials Appendix A) are described in the final version of the manuscript as shown in Supplemental Materials Appendix C.

Sensory evaluation of cosmetics questionnaire

The difficulties identified in the pretest of the sensory assessment instrument (Supplemental Materials Appendix B) are summarized in Table 4.

During the evaluation, the participants identified difficulties in conceptualizing the terms consistency/texture, absorption time, stickiness, and oiliness. To enable a better understanding of the terms, the interviewees suggested the inclusion of the terms “viscous” and “fluid” for better characterization of the attributes “consistency” and “texture” and the inclusion of the expression “drying time” for a clearer description of the attribute “absorption time”. In addition, all sensory parameters were defined in the question header.

Regarding the conceptual difficulties, in Question 6, one of the interviewees found that the alternative “neither weak nor strong” was defined as a negative attribute, when in fact the expression means only indecision on the part of the respondent. Thus, this item was replaced by “appropriate”, transmitting as well as a more impartial adjective.

As observed in Supplemental Materials Appendix B, from questions 3 to 16, the same attribute was evaluated from the interviewee’s opinion with a 9-point scale and from the sensory assessment of this attribute with intensity scales. However, some participants pointed out that they could not differentiate the conditions of references of these two forms of evaluation for the same attribute, indicating a difficulty of conditional character. This same category was identified in question 11. Nonetheless, the interviewee reported not understanding whether the question was a personal and hedonic opinion of the attribute or whether it should be evaluated as if there was an exact and more correct answer. To solve the conditional problems, the question header about the participant’s personal evaluation of the attribute was restructured.

Regarding scale misunderstandings, the participants reported difficulty in understanding both the 7-point and 9-point scale levels. It was observed that the interviewees were often marking only the extremes of the scale, not making use of intermediate levels. To work around this problem, explanations and numbers were added at each level to facilitate their general understanding.

All changes made in Supplemental Materials Appendix B are described in the final version of the version as shown in Supplemental Materials Appendix D.

 

Discussion

For the questionnaires to present consistent and reliable results, several factors must be considered during their development, such as the social, cultural, and psychological aspects of the target audience. In addition, they should be elaborated in a simple, easy-to-understand manner, avoiding ambiguous questions (19,20).

With the purpose of these requirements being met, the questionnaire should be pretested in a heterogeneous public (but within the target public), with people of different age groups and socio-educational profile in order to know what these characteristics influence in the understanding during the execution of the questionnaire (21). In this way, possible problems and inconsistencies in its completion can be detected, minimizing difficulties during the development of the study (22,23).

This work described five categories of problems that can be encountered during the elaboration of a questionnaire to substantiate cosmetic claims, as well as some options that can solve them.

The semantic validation is essential to ensure the correct understanding of all items by the interviewees, avoiding divergent interpretations of the intention expressed by the researcher. To this end, it is necessary that this step considers the relevance and coherence of all elements for the population to which the instrument is directed (24,25). In this study, it is noted that these semantic difficulties represent an unfamiliarity of technical level, making it impossible to differentiate the particularities of each term; thus, it is important that all attributes evaluated are defined and explained in each question.

Although the interviewee may know the technical definition of the attributes questioned, another difficulty identified was clinical self-assessment. For the questionnaire information to be provided more reliably, it is important that the interviewee has already received a previous diagnosis of the clinical condition evaluated. However, self-assessment can be used as an important tool for measuring the quality of life of participants (26,27).

Based on the analysis of the answers obtained, it was also observed that some participants did not understand the reference condition (conditional category) or the time and frequency (temporal category) at which the parameters should be evaluated. It is, therefore, necessary that the questions be drawn up in a clear, concrete, and precise way. The questions present in the instrument should also consider the reference system of the interviewee, his level of information, in addition to allowing a single interpretation and referring to one idea at time (18,28).

There are reports in the literature of the use of pretests in diverse areas, such as nutrition (22,29), social well-being (30,31), questionnaire adaptation in foreign languages (32,33), marketing and sales of the product (34), economy (35), and sociopolitical (36). There are similarities between this study and the pretest studies found in the literature, such as the need to explain technical terms or their exchange (29,32,35–37), clarify the use of scales and change them for a greater understanding (21,30,34). To assess the intensity of the attributes, numbers and verbal explanations were added to the intensity scales from 5 to 7 points (30). For the hedonic evaluation, the nine-point scale anchored with “Dislike extremely” and “like extremely” was maintained, because despite being more difficult to understand, when it comes to acceptance questions, the nine-point hedonic scale is considered the most reliable, valid and practical scale (4).

In addition, limitations were identified that can be harmful to the final result, such as the number of interviewees being lower than the number of people who will be interviewed with the final questionnaire (22). The importance of performing the pretest process was also scored within these articles, such as an easier understanding (27,33), as well as the fact that corrections prevent the results from being manipulated due to erroneously given answers (37).

 

Conclusions

The analysis of the pretest is very important to identify any types of errors that prevent the interviewee from fully understanding the instrument, and its correction is important so that the moment of response to the questionnaire does not become an unpleasant moment for the participant. The survey of the difficulties encountered, as well as the changes made to the data collection instrument, lead to a greater understanding of the tool by the respondents, and to the development of a more effective and reliable tool for substantiation of cosmetic claims.

The methodology of how to carry out a pre-test, its applicability and its improvement was also described, which is of great applicability for the cosmiatry sector. The method used here can be used for other questionnaires and extended to the most diverse areas of studies involving humans.

 

Authors Contributions Statement

MMV, conceptualization and study design; data analysis; drafting, editing and reviewing; LMA, conceptualization and study design; experimental implementation; drafting, editing and reviewing; TA, data analysis; drafting, editing and reviewing; GRL, conceptualization and study design; supervision and final writing.

 

Funding

This study was financed by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES No. 88887.489753/2020-00), and the São Paulo Research Foundation (FAPESP No. 2020/08516-0).

 

Acknowledgements

The authors thank the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), the São Paulo Research Foundation (FAPESP), and the National Council for Scientific and Technological Development (CNPq) for their financial support.

 

Conflict of Interests

The authors declare there are no financial and/or personal relationships that could present a potential conflict of interests.

 

References

  1. Fibrich, B. D., & Lall, N. (2018).Fighting the Inevitable: Skin Aging and Plants. In Medicinal Plants for Holistic Health and Well-Being (pp. 77–115). Elsevier. https://doi.org/10.1016/B978-0-12-812475-8.00003-2
  2. ABIHPEC. (2018). Caderno de Tendências higiene pessoal, perfumaria e cosméticos 2019-2020. https://abihpec.org.br/publicacao/caderno-de-tendencias-2019-2020/
  3. ABIHPEC. (2020). The future of the beauty market: new habits brought about by the pandemic should remain high even after the crisis. ABIHPEC - Brazilian Association of the Personal Hygiene, Perfumery and Cosmetics Industry. https://abihpec.org.br/o-futuro-do-mercado-de-beleza-novos-habitos-trazidos-pela-pandemia-devem-continuar-em-alta-mesmo-depois-da-crise/
  4. Guidelines for cosmetic product claim substantiation - Revising and expanding the Colipa Guidelines on Efficacy. (2019). Cosmetics Europe - The Personal Care Association. https://cosmeticseurope.eu/library/
  5. Galdorfini, B., de Almeida, M. G. J., Antonio, M., & Borges Isaac, V. L. (2012).Cosmetics’ Quality Control. In Latest Research into Quality Control. InTech. https://doi.org/10.5772/51846
  6. Vergilio, M. M., de Freitas, A. C. P., & da Rocha‐Filho, P. A. (2022).Comparative sensory and instrumental analyses and principal components of commercial sunscreens. Journal of Cosmetic Dermatology, 21(2), 729–739. https://doi.org/10.1111/jocd.14113
  7. Estanqueiro, M., Amaral, M. H., & Sousa Lobo, J. M. (2016).Comparison between sensory and instrumental characterization of topical formulations: impact of thickening agents. International Journal of Cosmetic Science, 38(4). https://doi.org/10.1111/ics.12302
  8. Lucian, R., & Dornelas, J. S. (2015).Mensuração de Atitude: Proposição de um Protocolo de Elaboração de Escalas. Revista de Administração Contemporânea, 19(spe2). https://doi.org/10.1590/1982-7849rac20151559
  9. Malhotra, N. (2012). Pesquisa de marketing: uma orientação aplicada(6aedição). Bookman.
  10. Gil, A. C. (2002). Como elaborar projetos de pesquisa(4aedição). Atlas.
  11. Campos, C. J. G. (2004). Content analysis: a qualitative data analysis tool in health care. Revista Brasileira de Enfermagem, 57(5), 611–614. https://doi.org/10.1590/S0034-71672004000500019
  12. Moraes, R. (1999). Análise de Conteúdo. Revista Educação, 22(37), 7–32.
  13. Domicio, I., Souza, S., Fuzio, T. S., & Takahashi, V. P. (2013). Estudo exploratório da percepção dos consumidores de cosméticos para os cuidados com a pele. XXI Congresso Latino Americano e Ibérico de Químicos Cosméticos, 1–13.
  14. ASTM.(2019). Standard Guide for Sensory Claim Substantiation. In A. International (Ed.), Annual Book of ASTM Standards(Vol. 00, Issue Reapproved, pp. 1–5). West Conshohocken. https://doi.org/10.1520/G0161-00R06.2
  15. Civille, G. V., Carr, B. T., & Meilgaard, M. C. (2016). Sensory Evaluation Techniques(5th ed).CRC Press.
  16. Mello, L. S. S., Almeida, E. L., & Melo, L. (2019).Discrimination of sensory attributes by trained assessors and consumers in semi-sweet hard dough biscuits and their drivers of liking and disliking. Food Research International. https://doi.org/10.1016/J.FOODRES.2019.01.031
  17. Lim, J. (2011). Hedonic scaling: A review of methods and theory. Food Quality and Preference, 22(8), 733–747. https://doi.org/10.1016/J.FOODQUAL.2011.05.008
  18. Bardin, L. (2015). Análise de Conteúdo(70 Edições (Ed.); 1st ed.).Almedina Br.
  19. Segot-Chicq, E., Compan-Zaouati, D., Wolkenstein, P., Consoli, S., Rodary, C., Delvigne, V., Guillou, V., & Poli, F. (2007). Development and validation of a questionnaire to evaluate how a cosmetic product for oily skin is able to improve well-being in women. Journal of the European Academy of Dermatology and Venereology, 0(0), 070807043105001-??? https://doi.org/10.1111/j.1468-3083.2007.02193.x
  20. Henkel, K. (2017). A categorização e a validação das respostas abertas em surveys políticos. Opiniao Publica, 23(3), 786–808. https://doi.org/10.1590/1807-01912017233786
  21. Sardinha, A., Levitan, M. N., Lopes, F. L., Perna, G., Esquivel, G., Griez, E. J., & Nardi, A. E. (2010). Tradução e adaptação transcultural do Questionário de Atividade Física Habitual. Archives of Clinical Psychiatry (São Paulo), 37(1), 16–22. https://doi.org/10.1590/S0101-60832010000100004
  22. Carvalho, F. S., Van Laer, N. M., Sachs, A., de Salvo, V. L. M. A., Coelho, L. de C., dos Santos, G. M. S., Akutsu, R. de C., & Asakura, L. (2010). Desenvolvimento e pré-teste de um questionário de frequência alimentar para graduandos. Revista de Nutricao, 23(5), 847–857. https://doi.org/10.1590/S1415-52732010000500015
  23. Perneger, T. V., Courvoisier, D. S., Hudelson, P. M., & Gayet-Ageron, A. (2015).Sample size for pre-tests of questionnaires. Quality of Life Research, 24(1), 147–151. https://doi.org/10.1007/s11136-014-0752-2
  24. Fuzissaki, M. de A., Santos, C. B. dos, Almeida, A. M. de, Gozzo, T. O., & Clapis, M. J. (2016). Validação semântica de instrumento para identificação da prática de enfermeiros no manejo das radiodermatites. Revista Eletrônica de Enfermagem, 18, e1142. https://doi.org/10.5216/ree.v18.35164
  25. Silva, L.M., Surniche, C.D., Sicsú, A.N., Mitano, F., Nogueira, J.D., Santos, C.B., Cunha, F.T., & Palha, P.F. (2015). Elaboração e validação semântica de um instrumento de avaliação da transferência do tratamento diretamente observado como política de controle da tuberculose. Revista Panamericana De Salud Publica-pan American Journal of Public Health,38, 129-135.
  26. Grob, J.J., Auquier, P., Martin, S., Lançon, C., & Bonerandi, J.J. (1999). Development and Validation of a Quality of Life Measurement for Chronic Skin Disorders in French: VQ-Dermato. Dermatology, 199,213 - 222.
  27. Lob-Corzilius, T., Böer, S., Scheewe, S., Wilke, K., Schon, M., Schulte im Walde, J., Diepgen, T.L., Gieler, U., Staab, D., Werfel, T., Schmid‐Ott, G., Fartasch, M., Wittenmeier, M., Schnopp, C., Kupfer, J., v. Schlippe, A., Szczepanski, R., & Keins, P. (2004). The ‘Skin Detective Questionnaire’: A Survey Tool for Self-Assessment of Patients with Atopic Dermatitis. First Results of Its Application. Dermatology and Psychosomatics / Dermatologie und Psychosomatik, 5,141 - 146.
  28. Jones, T. L., Baxter, M. A., & Khanduja, V. (2013). A quick guide to survey research. Annals of the Royal College of Surgeons of England, 95(1), 5–7. https://doi.org/10.1308/003588413X13511609956372
  29. Oliveira, L., Poínhos, R., Sousa, F. & Silveira, M. G(2016) Construção e Validação de um Questionário para Avaliação da Perceção sobre Alimentos Funcionais. Acta Portuguesa de Nutrição. 7, 14-17.https://doi.org/10.21011/apn.2016.0704
  30. Hilton C. E. (2017) The importance of pretesting questionnaires: a field research example of cognitive pretesting the Exercise referral Quality of Life Scale (ER-QLS).International Journal of Social Research Methodology 20, 21–34. https://doi.org/10.1080/13645579.2015.1091640
  31. Del Castillo, L., Leporace, G., Cardinot, T., Levy, R.A., Oliveiria,L.P. (2012) A importância dos questionários para avaliação da qualidade de vida. Revista Hospital Universitário Pedro Ernesto, UERJ, 11 (1), 12-17.
  32. Chor D, Alves M G de M, Giatti L et al. (2013) Questionario do ELSA-Brasil: desafios na elaboracao de instrumento multidimensional. Revista Saude Publica 47(Suppl 2), 27–36. https://doi.org/10.1590/S0034-8910.2013047003835.
  33. Marx, F. C., de Oliveira, L. M. Bellini, C.G., Ribeiro, M.C.C. (2006) Tradução e validação cultural do questionário algofuncional de Lequesne para osteoartrite de joelhos e quadris para a língua Portuguesa. Revista Brasileira de Reumatologia, 46, 253–260.
  34. Vuolle, M., Tiainen, M., Kallio, T., Vainio, T., Kulju, M., & Wigelius, H. (2008).Developing a questionnaire for measuring mobile business service experience. Mobile HCI.
  35. Dikko, M. (2016). Establishing Construct Validity and Reliability: Pilot Testing of a Qualitative Interview for Research in Takaful (Islamic Insurance). The Qualitative Report, 21(3), 521-528. https://doi.org/10.46743/2160-3715/2016.2243
  36. Alhujran, O. Determinants of e-government services adoption in developing countries: a field survey and a case study, Doctor of Philosophy thesis, School of Information Systems and Technology, Faculty of Informatics, University of Wollongong, 2009. https://ro.uow.edu.au/theses/1998
  37. Moreira, R. S., Magalhães, L de C, Siqueira, C. M., Alves, C.R.L. (2019) Cross-cultural adaptation of the child development surveillance instrument “Survey of Wellbeing of Young Children (SWYC)” in the Brazilian context. Journal of Human Growth and Development 29, 28–38. http://dx.doi.org/10.7322/jhgd.145001