Congratulations on tackling such a large screening task! AI-assisted tools like ASReview and Rayyan can significantly streamline the process. Both use Active Learning, where the AI learns from your decisions and prioritizes articles it thinks are most likely to be relevant.

Here's a guide to setting up both, along with important considerations.

General Pre-Screening Steps (Crucial for ANY Tool)

Before you even touch ASReview or Rayyan, ensure these steps are completed:

• Define Clear Inclusion/Exclusion Criteria: This is the most critical step. Your criteria should be explicit and consistently applied. Draft them, pilot them on a small set of articles, and refine them.


• Combine & Deduplicate References:


• Gather all your references (5000 articles) from various databases (e.g., Scopus, Web of Science, ERIC, PsycINFO).


• Use a reference manager (Zotero, Mendeley, EndNote) or a dedicated deduplication tool (like a Systematic Review Deduplication tool, or even Excel/OpenRefine for advanced users) to remove duplicates before importing them into your screening tool. This is vital as AI tools won't deduplicate for you, and screening duplicates wastes time and effort.


• Export in a Compatible Format:


• RIS (.ris): Most commonly used and recommended for both tools as it contains rich metadata (title, abstract, authors, keywords).


• CSV (.csv): Ensure you have columns for "Title" and "Abstract" at a minimum.


• BibTeX (.bib): Also generally supported.

Option 1: Setting Up Rayyan for AI-Assisted Screening

Rayyan is a popular, web-based tool known for its user-friendliness and collaborative features. It's often free for academics.

Pros of Rayyan:

• Web-based: No installation, accessible from anywhere.


• Intuitive UI: Easy to learn and use, especially for beginners.


• Collaboration: Excellent features for multiple reviewers, blinding, and conflict resolution.


• Built-in AI: Automatically suggests articles based on your initial screening.


• Keyword Highlighting: Helps identify relevant terms.

• Proprietary (though free for academics).


• Less customizable AI model compared to ASReview.


• Requires an internet connection.

Rayyan Setup Steps:

• Create an Account:


• Go to rayyan.ai.


• Sign up for a free academic account using your institutional email.


• Create a New Review:


• Once logged in, click "New Review" or "Create New Review."


• Give your review a Title (e.g., "Educational Technology Effectiveness SR").


• Add a Description (optional, but good for context).


• You can set it as "Public" or "Private." For most systematic reviews, "Private" is recommended.


• Import References:


• Click "Add References."


• Drag and drop your deduplicated .ris, .csv, or .bib file(s) into the upload area, or browse to select them.


• Rayyan will process the files. If it detects potential duplicates (even after your initial deduplication), it might prompt you to review them. Proceed with import.


• Configure Screening Settings (Optional but Recommended):


• Blinding: By default, Rayyan sets up "blinding," meaning reviewers cannot see each other's decisions until the review stage. This is standard practice in systematic reviews. You can enable/disable it in the review settings.


• Inclusion/Exclusion Keywords: This is a powerful Rayyan feature.


• Go to "Settings" -> "Inclusion/Exclusion"


• Add keywords that, if present, strongly indicate inclusion or exclusion. Rayyan will highlight these in abstracts. Be careful not to over-rely on this for definitive decisions, but it's a great initial flag.


• Start Screening and Train the AI:


• Go to the "Screening" tab.


• Rayyan will present articles one by one. For each article, you'll see:


• Title


• Abstract


• Keywords (often highlighted by Rayyan's AI)


• You have three primary decisions: Include, Exclude, or Maybe.


• The AI (The "Suggest" Column): As you screen articles (especially the first 50-100), Rayyan's AI will start learning your patterns. Look for the "Suggest" column (it might appear as a little robot icon or a score).


• Green Checkmark: Rayyan suggests inclusion.


• Red X: Rayyan suggests exclusion.


• Question Mark: Rayyan is uncertain.


• Prioritization: Rayyan automatically prioritizes articles it thinks are most relevant at the top of your screening list, or you can sort by "Suggestions" to see its top picks.


• Keep Screening: Continue screening, paying attention to the AI's suggestions. The more you screen, the smarter Rayyan's AI becomes. Aim to screen at least 10% of your total articles, or until you feel the AI is consistently making good suggestions.


• Collaborate (If Applicable):


• Go to "Reviewers" -> "Invite Reviewers."


• Enter their email addresses. They will need a Rayyan account.


• Rayyan manages dual screening and conflict resolution very well.

Option 2: Setting Up ASReview for AI-Assisted Screening

ASReview (Active learning for Systematic Reviews) is an open-source, local-first tool that offers greater flexibility and control over the AI models.

Pros of ASReview:

• Open-Source & Free: Complete control, community-driven.


• Local Processing: Data stays on your computer (privacy), faster processing once set up.


• Customizable AI: You can choose different classifiers, query strategies, and feature extractors.


• Scalable: Excellent for very large datasets.


• Offline Access: Once installed, no internet needed for screening.

• Installation Required: Needs Python, can be a bit more technical.


• Local-only: Not inherently collaborative without external solutions (e.g., sharing project files).


• Slightly Steeper Learning Curve: The GUI is user-friendly, but understanding the underlying AI options helps.

ASReview Setup Steps:

• Install Python:


• ASReview requires Python (version 3.8 or higher is recommended).


• Download and install Python from python.org if you don't have it already.


• Important: During installation, make sure to check "Add Python to PATH" or "Add Python 3.x to path."


• Install ASReview:


• Open your computer's Command Prompt (Windows) or Terminal (macOS/Linux).


• Run the following command to install ASReview:

pip install asreview

• For the GUI (Recommended for Ease of Use): Install the GUI template:

pip install asreview-template-gui

• Wait for the installation to complete.


• Prepare Your Data:


• Ensure your deduplicated articles are in a .ris, .csv, or .bib file. .ris is generally preferred for its rich metadata.


• Launch the ASReview GUI:


• In your Command Prompt/Terminal, type:

asreview gui

• This will open a new tab in your web browser (usually at http://localhost:5000/). This is your local ASReview interface.


• Create a New Project:


• On the ASReview GUI, click "Start new review."


• Name: Give your project a descriptive name.


• Description: (Optional) Add details about your review.


• Import References:


• Click "Upload new dataset."


• Select your .ris (or .csv, .bib) file.


• ASReview will process and display a summary of the imported references.


• Configure AI Model (Crucial for Prioritization):


• This is where ASReview shines in customization. You'll see options for:


• Classifier: The machine learning algorithm that learns from your decisions.


• svm (Support Vector Machine): Often a good default for text classification.


• logistic (Logistic Regression)


• nb (Naive Bayes)


• Query Strategy: How the AI selects the next article to present to you.


• max (Maximum Certainty): Prioritizes articles the AI is most confident about (either highly relevant or highly irrelevant).


• uncertainty: Prioritizes articles the AI is least certain about. This can be good for exploring ambiguous cases.


• random (Random sampling)


• Feature Extractor: How the text (titles/abstracts) is converted into numerical data for the AI.


• tfidf (TF-IDF): Common and effective for weighting word importance.


• doc2vec


• Balance Strategy: How the model handles imbalanced datasets (e.g., many more irrelevant than relevant articles).


• undersampling, oversampling. Often none is fine for initial steps.


• Recommendation for starting:


• Classifier: svm


• Query Strategy: uncertainty (This often works well for active learning to explore the boundaries and make the AI smarter faster).


• Feature Extractor: tfidf


• Initial Training/Seed Articles:


• You'll be asked how many "initial relevant" and "initial irrelevant" articles to label. ASReview needs a few examples to start learning.


• You can manually label these from your dataset, or ASReview can pick them randomly. For 5000 articles, starting with 5-10 relevant and 5-10 irrelevant is a good starting point if you know some examples. If not, ASReview will pick them for you, and you'll label them first.


• Start Screening and Train the AI:


• Click "Start Review."


• ASReview will present articles. For each:


• Read the title and abstract.


• Click "Relevant" or "Irrelevant."


• You can also mark as "Not sure" if needed, though the AI benefits most from clear decisions.


• AI Prioritization: ASReview will continuously learn from your decisions. The list of "Unlabeled papers" will be reordered based on the AI's prediction scores. Articles the AI predicts as most likely to be relevant will appear higher in the list.


• Continue Screening: Keep screening. The AI will become more accurate with more training data.


• Monitor Progress:


• ASReview provides plots showing "Number of included papers" vs. "Number of screened papers," "Efficiency," and "Recall." These help you understand how much of the relevant literature you've likely found.

Key Considerations & Best Practices for AI-Assisted Screening:

• "Cold Start" Problem: Both tools need initial manual screening to train their AI. Expect the first 50-200 articles to be purely manual, and the AI's suggestions to be less reliable. Its performance will improve significantly as you screen more.


• Don't Blindly Trust the AI: The AI is a prioritization tool, not a decision-maker. Always critically review the articles it suggests. It can make mistakes, especially with nuanced criteria.


• Stopping Criteria: How do you know when to stop screening?


• Recall: ASReview's plots can show you when the curve of relevant articles starts to flatten, suggesting you've found most of them.


• Saturation: When you screen a large number of articles (e.g., 200-500 consecutively) and find very few or no new relevant ones.


• Percentage: After screening a certain percentage (e.g., 50-70%) of the total, or a fixed number of irrelevant articles in a row.


• Consistency: Maintain consistent application of your inclusion/exclusion criteria. If you have multiple screeners, calibrate thoroughly before starting and meet regularly to discuss discrepancies.


• Documentation: Keep detailed notes on your criteria, any changes, and your screening process. This is vital for transparency and reproducibility.


• Saving and Backup (ASReview specific): ASReview projects are saved locally. Regularly back up your project files (often in a hidden .asreview folder in your home directory or where you ran asreview gui).


• Computational Resources (ASReview specific): For 5000 articles, ASReview should run fine on most modern computers. For much larger datasets (tens of thousands), more RAM might be beneficial.