**The Work:**
1. **Summary:**
The creation of a “Research Proposal” for application to the PhD program at two universities in Europe: UOC – Open University of Catalonia (Spain) and UTAD – University of Trás-os-Montes and Alto Douro (Portugal).
2. **Desired Research Line at UOC:**
One of these universities (UOC) has a research line that aligns well with my intended research:
Are the texts written by large LLMs better than those of an average human?
Recent studies have found that readers prefer texts generated by LLMs over similar texts extracted from Wikipedia articles [1]; and that the results of crowdsourced text summarization tasks are of lower quality compared to those obtained with LLMs [2]. To explore and contrast these findings in a broader context, we propose designing tasks and experiments to analyze the quality of texts written by LLMs like ChatGPT in relation to those written by humans. Specifically, we aim to explore research questions such as: What percentage of humans are still able to write better texts than LLMs? How do these percentages vary depending on the task and type of text? Can we use LLMs to rate the quality of these texts and achieve similar judgments as human evaluators?
[1] Huschens, M., Briesch, M., Sobania, D., & Rothlauf, F. (2023). Do You Trust ChatGPT?—Perceived Credibility of Human and AI-Generated Content. arXiv preprint arXiv:2309.02524.
[2] Veselovsky, V., Ribeiro, M. H., & West, R. (2023). Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks. arXiv preprint arXiv:2306.07899.
3. **Research Proposal Idea:**
The idea is to extract over 170,000 articles from a specific domain of knowledge from the website wol.jw.org. This site is a library of religious articles from the Jehovah’s Witnesses denomination since 1950. The selection of articles will cover a diverse range of topics to enhance the robustness of the training data. After extracting these documents, the plan is to train an LLM with specific guardrails and ethical guidelines so that it can respond as accurately as possible within the specified knowledge area.
Post-training, a series of quantitative and qualitative tests will be conducted on the responses generated by the LLM, including comparing religious questions answered by the LLM to those answered by leaders of this religious denomination (pastors or elders). One of the ideas is to submit the responses generated by pastors and the LLM to ordinary religious individuals (neither pastors nor elders) in a blind test and rank them on a Likert scale.
Evaluation metrics will include coherence, relevance, and adherence to doctrinal accuracy, using both automated evaluation tools and human judgment. The Research Proposal should include an ethical analysis of creating and using an LLM for “spiritual guidance,” answering personal questions about relationships, quality of life, principles, etc.
4. **Assumptions:**
– **Alignment with Research Lines:**
Clearly articulate how the proposed research aligns with the research lines of both universities, emphasizing its novelty and relevance. Highlight any potential interdisciplinary aspects that could appeal to a broader academic audience.
– **Additional Tests and Research Questions:**
Propose additional tests that could provide further insights, such as longitudinal studies to observe changes in user perceptions over time. Suggest supplementary research questions that explore related topics, such as the impact of cultural differences on the acceptance of LLM-generated texts.
– **Comprehensive Literature Review:**
Include a thorough literature review with at least six additional references that provide context and support for the proposed research. Ensure the references are up-to-date and cover a range of perspectives, including previous studies on LLMs, text quality evaluation, and ethical considerations in AI. Use APA format for all references.
– **Collaborative Opportunities:**
Identify potential collaborators within the universities who have expertise in relevant areas, such as natural language processing, ethics, or religious studies. Highlight any opportunities for joint research projects or cross-departmental initiatives that could enhance the proposal’s appeal.
– **Practical Applications:**
Discuss the potential practical applications of the research findings, such as improving LLM-based tools for educational or counseling purposes. Emphasize how the research could contribute to broader discussions on AI ethics and the responsible use of technology in sensitive domains.