Accelerating Ecotoxicology Reviews: A 2025 Guide to Automated Screening Tools and AI Workflows

Abstract

Systematic reviews in ecotoxicology face unique challenges, including interdisciplinary terminology, diverse study methodologies, and a vast, growing literature base. This article provides a comprehensive guide for researchers and scientists on leveraging digital tools to automate the labor-intensive screening phase. We explore the foundational need for automation, detail the application of leading software and AI-assisted methods, address practical troubleshooting and optimization strategies, and present comparative validations of current technologies. The goal is to empower research teams to conduct more efficient, transparent, and reproducible evidence syntheses, ultimately accelerating the integration of toxicological evidence into environmental and biomedical decision-making.

Why Automate? Defining the Need and Tools for Ecotoxicology Systematic Reviews

Technical Support Center: Troubleshooting Systematic Review Screening

This technical support center addresses common operational challenges researchers face when screening literature for systematic reviews (SRs) in ecotoxicology. The guidance is framed within a thesis exploring tools for automating this screening process, focusing on overcoming hurdles posed by interdisciplinary jargon and diverse methodologies [1].

Frequently Asked Questions (FAQs)

Q1: Our screening process is overwhelmed by the volume of papers from different fields (e.g., chemistry, ecology, hydrology). How can we manage this complexity efficiently? A: The volume and diversity are central challenges in ecotoxicology [1]. Implement a structured screening workflow and leverage AI-assisted tools designed for multi-disciplinary corpora. Begin by using a platform like Sysrev, which introduces machine learning to increase the accuracy and efficiency of the review process [2]. For very large datasets, consider a multi-agent AI system like InsightAgent, which partitions the literature corpus based on semantic similarity, allowing parallel processing of different disciplinary clusters [3].

Q2: Reviewers from different disciplines interpret the same eligibility criteria differently, leading to inconsistencies. How can we standardize screening? A: This is a known issue where interdisciplinary terminology leads to variable interpretation [1]. The solution is a three-step protocol: First, hold calibration meetings to develop a unified, written glossary of key terms (e.g., bioavailability, LC50, biomagnification) [4] [5]. Second, translate these agreed-upon eligibility criteria into a precise prompt for a fine-tuned Large Language Model (LLM) [1]. Third, use the AI model to perform a first-pass screening on all articles, ensuring a consistent application of the baseline criteria, which human reviewers can then verify.

Q3: We are considering an AI tool for screening. What are the critical performance metrics, and what accuracy can we realistically expect? A: The critical metrics are recall (sensitivity) and precision, and their harmonized measure, the F1 score. Agreement with human experts is typically measured using Cohen's Kappa for two raters or Fleiss' Kappa for multiple raters [1]. Realistic performance varies by task: a recent AI agent system demonstrated a 47% improvement in F1 score for article identification with user interaction [3]. Another study using a fine-tuned ChatGPT model reported "substantial agreement" at the title/abstract stage and "moderate agreement" at the full-text stage compared to human reviewers [1]. Expect to iteratively refine the AI model with expert feedback to achieve optimal results.

Q4: How do we choose the right database or knowledgebase for ecotoxicology data extraction after screening? A: For curated ecotoxicology data, the EPA ECOTOX Knowledgebase is an essential resource. It contains over one million test records for more than 12,000 chemicals and 13,000 species [6]. Use its advanced SEARCH and EXPLORE features to filter data by specific endpoints, species, and test conditions. For human exposure assessment data to support risk assessment, refer to systematic scoping reviews that identify and evaluate accessible computational tools and models [2].

Troubleshooting Guides

Problem: Low Inter-Rater Reliability (IRR) During Manual Title/Abstract Screening Symptoms: Low Cohen's Kappa scores among reviewers, frequent disagreements during consensus meetings, unpredictable inclusion/exclusion decisions. Diagnosis: Inconsistent application of eligibility criteria due to ambiguous terminology or a lack of shared understanding of interdisciplinary concepts. Solution:

• Pause Screening & Refine Criteria: Hold a dedicated calibration workshop with the review team.
• Create a Decision Tree: Visually map the inclusion/exclusion criteria with specific examples from pilot articles.
• Develop a Disambiguation Glossary: Build a living document defining ambiguous terms (e.g., "adverse effect," "chronic exposure," "model") as they are used in the context of your review [4] [5].
• Pilot Test Revised Protocol: Screen a new batch of 50-100 articles independently. Calculate IRR again. Repeat steps 1-4 until IRR reaches an acceptable level (e.g., Kappa > 0.6).
• Implement AI-Assisted Consistency Check: Use a fine-tuned LLM to screen the same batch and compare its decisions to the reconciled human decisions to identify any remaining systemic ambiguities in the criteria [1].

Problem: Poor Recall or Precision from an AI Screening Tool Symptoms: The AI model is missing too many relevant papers (low recall) or including too many irrelevant ones (low precision). Diagnosis: The model has not been adequately trained or fine-tuned on domain-specific, labeled data representative of your research question. Solution:

• Audit the Training Data: Ensure your labeled dataset (used for fine-tuning) is of high quality, balanced, and reflects the interdisciplinary scope of your review.
• Optimize Hyperparameters: Adjust the model's technical settings. For an LLM like GPT, key parameters include:
  • Temperature (0.1-0.5): Lower for more deterministic, consistent outputs during screening.
  • Top_p (0.8-0.95): Controls the diversity of predicted tokens.
  • Epochs (3-10): Increase if the model is underfitting, decrease if it is overfitting [1].
• Implement a Multi-Agent or Ensemble Approach: If using a single agent, consider switching to a framework like InsightAgent, which uses multiple AI agents to process different semantic clusters of literature in parallel, improving overall accuracy [3].
• Integrate Human-in-the-Loop Feedback: Use an interactive platform where human experts can correct the AI's screening decisions in real-time. This feedback should be used to continuously re-train and improve the model.

Problem: Difficulty Synthesizing Findings from Methodologically Diverse Studies Symptoms: Inability to perform meaningful meta-analysis, qualitative synthesis feels fragmented, results from different study types (e.g., field monitoring vs. lab microcosms) appear contradictory. Diagnosis: This is a fundamental challenge in interdisciplinary ecotoxicology reviews [1]. The screening phase did not adequately categorize studies by methodology for later synthesis. Solution:

• Tag During Screening: During the full-text screening phase, implement additional tagging for key methodological variables (e.g., study_type: field_observational, lab_experimental, computational_model; scale: mesocosm, watershed, population).
• Use a Structured Data Extraction Tool: Employ tools that force extraction into predefined fields related to methods (e.g., test organism life stage, exposure duration, measured endpoints) rather than free text. The ECOTOX Knowledgebase data structure is a good model [6].
• Synthesize by Methodological Group: Structure your results synthesis not just by outcome, but by methodological approach. Clearly state how different methods contribute to the overall weight of evidence, acknowledging the strengths and limitations of each [7].

Experimental Protocols for Cited Screening Methodologies

• Objective: To consistently apply interdisciplinary eligibility criteria for title/abstract screening in an SR.
• Materials: Access to OpenAI API (GPT-3.5 Turbo or similar), a corpus of article titles/abstracts in a manageable format (CSV/JSON), a labeled training dataset of 100-200 articles reviewed by experts.
• Procedure:
  • Expert Calibration: Reviewers independently screen a pilot set of articles, resolve disagreements, and finalize eligibility criteria.
  • Prompt Engineering: Translate the final eligibility criteria into a clear, structured LLM prompt (e.g., "You are a systematic review screener. Based on the following title and abstract, determine if the study is relevant. Criteria: [List]. Respond only with 'Include' or 'Exclude.'").
  • Model Fine-Tuning: Use the OpenAI fine-tuning API with your labeled dataset. Suggested hyperparameters: epochs=4, learning_rate_multiplier=0.1, batch_size=8 [1].
  • Stochastic Sampling & Decision: Run the fine-tuned model on each article multiple times (e.g., 15 runs) due to inherent stochasticity. Take the majority vote as the final decision [1].
  • Validation: Apply the model to a held-out test set of expert-screened articles. Calculate agreement metrics (Cohen's Kappa, F1 score).

• Objective: To rapidly screen and synthesize a very large, multidisciplinary corpus for an SR.
• Materials: InsightAgent or similar multi-agent framework, full-text or abstract corpus.
• Procedure:
  • Corpus Mapping & Partitioning: The system projects the literature corpus into a Radial-based Relevance and Similarity (RSS) map, where article position indicates relevance (center) and semantic similarity (clusters) [3].
  • Cluster Assignment: The map is partitioned into distinct semantic clusters (e.g., using K-means). Each cluster is assigned to a dedicated AI agent.
  • Parallel Agent Processing: Each agent explores its cluster, starting from the most relevant (central) articles. It reads, summarizes, and makes screening decisions based on the review's objectives.
  • Human Oversight & Interaction: Researchers monitor the agents' "trajectories" on the visual map. They can intervene to redirect an agent, adjust its focus, or correct its decisions, providing real-time expert feedback.
  • Evidence Synthesis: Agents generate interim summaries for their clusters. A final synthesis agent or the human team integrates these into a cohesive review, supported by a provenance tree tracing claims back to source articles [3].

• Objective: To conduct a broad scoping review of available methods and tools (e.g., for exposure assessment).
• Materials: Sysrev web platform or similar (DistillerSR, Rayyan), predefined PICOS (Population, Intervention, Comparator, Outcome, Study) criteria.
• Procedure:
  • Platform Setup: Upload retrieved references to the platform. Define inclusion/exclusion fields and screening forms.
  • AI-Assisted Priority Screening: Screen an initial random subset of references (e.g., 3,000) with the platform's AI "watching." The AI learns to predict the likelihood of inclusion.
  • Prioritized Screening: Screen the remaining references in order of the AI-predicted likelihood of inclusion (e.g., >45% likelihood). This increases the rate of finding relevant papers quickly [2].
  • Dual Verification: Maintain a human verification step, especially for lower-probability articles, to ensure recall.
  • Data Extraction & Mapping: Use the platform's tools to extract and chart key data (e.g., tool names, chemical classes, exposure routes) from included studies to map the available evidence [2].

Table 1: Performance Comparison of AI Screening Methodologies

Methodology	Reported Efficiency Gain	Key Strength	Primary Challenge	Best Suited For
Fine-Tuned LLM [1]	Substantial agreement with humans (Kappa)	High consistency in applying complex criteria	Requires quality labeled data for tuning	Reviews with clear, complex eligibility rules
Multi-Agent AI (InsightAgent) [3]	Completes SR in ~1.5 hours vs. months	Handles large, diverse corpora via parallel processing	System complexity; requires interactive oversight	Large, interdisciplinary reviews
AI-Prioritized Screening (Sysrev) [2]	Increased relevant hit rate during screening	Efficiently prioritizes workload for human screeners	Less autonomous; still human-dependent	Scoping reviews & large-scale evidence mapping

Visualizations of Screening Workflows and Relationships

Diagram 1: Ecotoxicology Systematic Review Screening Workflow (Max Width: 760px)

Diagram 2: AI-Human Collaborative Screening System Architecture (Max Width: 760px)

The Scientist's Toolkit: Research Reagent Solutions

Table 2: Essential Digital Tools & Platforms for Ecotoxicology Review Screening

Tool/Resource Name	Type	Primary Function in Screening	Key Consideration
Sysrev [2]	Web Platform	Integrates machine learning to prioritize and manage the screening process for systematic/scoping reviews.	Effective for evidence mapping and reviews with clear, categorical inclusion data.
InsightAgent [3]	Multi-Agent AI Framework	Partitions literature by semantics for parallel AI agent processing, with human-in-the-loop visualization.	Designed for rapid synthesis of large corpora; requires technical setup and interactive oversight.
Fine-Tuned LLM (e.g., GPT) [1]	AI Model	Provides consistent, automated application of complex eligibility criteria to titles/abstracts/full texts.	Performance depends heavily on the quality of training data and prompt engineering.
EPA ECOTOX Knowledgebase [6]	Curated Database	Provides pre-extracted toxicity data for chemicals and species; useful for validating scope and informing criteria.	Not a screening tool per se, but a critical resource for defining relevant endpoints and understanding data landscape.
Explainable AI (XAI) Principles [8]	Conceptual Framework	Guides the selection and implementation of AI tools that provide transparent, interpretable decisions for auditing.	Critical for maintaining scientific rigor and trust when using "black box" AI models in high-stakes reviews.
Interdisciplinary Glossary [4] [5]	Documentation	Serves as an agreed-upon reference to align team understanding of key toxicological and ecological terms.	A simple but foundational tool to mitigate the core challenge of interdisciplinary jargon [1].

Technical Support Center: Troubleshooting Automated Screening Tools

Frequently Asked Questions (FAQs)

Q1: Our automated screening tool (e.g., ASReview, Rayyan with AI) is flagging too many irrelevant studies in the 'included' set after the first training round. What went wrong? A: This is often due to unrepresentative or insufficient initial training data. The algorithm may be overfitting to your first few relevance judgments.

• Solution: Re-initialize the screening and provide a more balanced "prior knowledge" set. Manually identify and include 5-10 highly relevant ("golden") studies and 10-15 clearly irrelevant studies before starting active learning. This teaches the model the boundaries of your inclusion criteria more effectively.

Q2: During dual-reviewer screening with an AI-assisted tool, how do we resolve discrepancies when the AI's prediction heavily influenced one reviewer? A: The AI should be an aid, not an arbitrator. Implement a blinded reconciliation phase.

• Protocol: 1) Both reviewers screen independently with AI suggestions hidden. 2) Compare results. 3) For conflicting decisions, reveal the AI's prediction score and the other reviewer's decision. 4) Discuss the study's abstract/full text against the protocol's PICO criteria to make a final, consensus decision. This minimizes automation bias.

Q3: We are using a text classifier (e.g., in DistillerSR, SWIFT-Review) and performance seems poor for our ecotoxicology topic. How can we improve it? A: Ecotoxicology-specific terminology may not be well-represented in general models.

• Solution: Create and apply a custom synonym dictionary or thematic lexicon. For example, map all variant terms ("Daphnia magna," "D. magna," "water flea") to a standardized key. Augment your training data by including studies from known, relevant systematic reviews in your field to improve contextual understanding.

Q4: Our screening workflow keeps stalling at the deduplication stage, with many false positives. A: Standard deduplication often fails with preprints, conference abstracts, and different database export formats.

• Troubleshooting Guide:
  • Pre-Process Exports: Ensure all imports are in the same format (e.g., RIS) and from consistent sources.
  • Use Fuzzy Matching: Enable settings for matching on "Title + Author + Year" with a similarity threshold (e.g., 90-95%).
  • Manual Check Pass: Sort the "duplicate groups" by confidence score and manually verify the top 50-100 groups. This trains you to identify common false-positive patterns (e.g., series reports).
  • Protocol Note: Document your exact deduplication settings (software, fields, algorithm) in your methods section for reproducibility.

Q5: How do we validate that our AI-assisted screening process did not miss key studies? A: You must perform a validation check, often called a "stopping rule" verification.

• Detailed Methodology:
  • After the AI-assisted screening is complete (e.g., after screening 20% of the corpus), take a random sample of all records excluded by the tool.
  • The sample size should be statistically justified (e.g., 95% confidence level, 5% margin of error). For 10,000 excluded records, sample ~370.
  • Screen this sample of "excluded" studies at the full-text level against your eligibility criteria.
  • Calculate the proportion of missed relevant studies. If this proportion is below an acceptable threshold (e.g., <1%), you can confidently stop. If a relevant study is found, retrain the model and continue screening.

---

Quantitative Data on the Screening Burden

Table 1: Time and Cost Implications of Manual vs. Automated Screening

Metric	Manual Screening (Traditional)	AI-Assisted Screening (Active Learning)	Data Source & Context
Screening Time	100% (Baseline)	Reduced by 50-90%. Typically requires screening only 10-25% of the total corpus to identify 95% of relevant studies.	- Simulation studies across biomedical domains.
Cost Per Review	High. Primarily driven by personnel time (weeks to months of salary).	Substantially Lower. Reduces person-hours by a proportional amount to time saved. Software costs are fixed.	- Economic analyses of systematic review production.
Human Error Rate (Missed Studies)	Estimated 5-10% inconsistency rate between independent human reviewers.	Can be reduced to <1-2% when used with a proper validation stop rule (see FAQ #5).	- Studies on inter-rater reliability in environmental health reviews.
Optimal Use Case	Necessary for very small datasets (<100 records) or when criteria are highly complex and non-textual.	Essential for large-scale reviews (>1000 records). Most beneficial in the title/abstract phase.	Best practice guidelines from CEEDER, SRDB.

---

Experimental Protocol: Implementing an AI-Assisted Screening Workflow

Title: Protocol for a Dual-Reviewer, AI-Powered Title/Abstract Screening Phase in Ecotoxicology.

Objective: To efficiently and accurately screen a large bibliographic dataset (n>5000) for relevance to a predefined PICO question using active learning.

Materials: See "The Scientist's Toolkit" below.

Methodology:

• Preparation:
  • Develop and register the systematic review protocol (including PICO).
  • Export search results from all databases into RIS files.
  • Import all files into the chosen screening tool (e.g., ASReview).
  • Perform automated deduplication using fuzzy matching on title, author, and year.

• Prior Knowledge Injection (Critical Step):

  • The review team manually identifies and labels 5-10 key relevant studies and 10-15 clearly irrelevant studies from the corpus. These are loaded into the tool as the initial training set.

• Active Learning Screening Loop:

  • The AI model (e.g., naive Bayes, SVM) is trained on the initial set.
  • The tool presents one record at a time, ranked by its predicted probability of relevance.
  • Two independent reviewers screen each presented record, blinded to each other's decision and the AI's prediction score. They label each as "relevant" or "irrelevant."
  • Each new decision is added to the training data, and the model is updated in real-time.
  • This loop continues.

• Stopping Rule & Validation:

  • A stopping rule is pre-defined (e.g., after 100 consecutive irrelevant records).
  • Upon triggering the rule, execute the validation methodology described in FAQ #5.
  • If the validation passes, the title/abstract phase is complete. Proceed to full-text retrieval.

• Reconciliation:

  • For records where reviewers disagreed during the active learning phase, a third reviewer adjudicates based on the PICO criteria, with the AI score remaining hidden.

---

Visualizations

Diagram 1: AI-Assisted Systematic Review Workflow

Diagram 2: Human-AI Interaction in Screening Decision

---

The Scientist's Toolkit: Research Reagent Solutions

Table 2: Essential Tools for Automated Screening in Ecotoxicology

Tool / Resource	Function & Explanation
ASReview (Open Source)	Core active learning screening software. Allows for custom model selection and is highly flexible for research on screening automation itself.
Rayyan (Freemium)	Web-based tool with a user-friendly interface and basic AI assistance. Excellent for collaborative screening across institutions.
DistillerSR (Commercial)	Full-featured, enterprise-level systematic review management software with advanced AI, deduplication, and workflow customization.
SYRCLE's Toolbox	A set of tools and guidelines specifically for animal studies, crucial for adapting PICO criteria for ecotoxicology models.
EndNote / Zotero	Reference managers for initial collection and deduplication before import into specialized screening tools.
PubMed / ETOX DB APIs	Programmatic access to database entries allows for reproducible search strategies and bulk data retrieval.
Custom Ecotoxicology Lexicon	A pre-defined list of standardized terms (species, chemicals, endpoints) to improve text mining accuracy.
Reporting Guideline (PRISMA)	The PRISMA checklist and flow diagram template are essential for reporting the modified, AI-assisted screening method transparently.

This technical support center is designed for researchers, scientists, and drug development professionals conducting systematic reviews in ecotoxicology. It provides targeted troubleshooting guides and FAQs to help you overcome common challenges when implementing automation tools for screening studies. The content is framed within a broader thesis on enhancing the efficiency and reliability of evidence synthesis through technological innovation [9] [10].

Troubleshooting Workflow for Automation Tools

Adopting a structured approach is critical when diagnosing issues with systematic review automation. The following workflow, adapted from established technical troubleshooting methodologies, provides a logical progression from problem identification to resolution [11].

Frequently Asked Questions (FAQs) and Troubleshooting Guides

Tool Selection and Implementation

• Problem: I'm overwhelmed by the number of tools available (e.g., Covidence, Rayyan, DistillerSR). How do I choose the right one for my ecotoxicology review? [9] [12]

  • Solution: Base your selection on project needs. For a full-process platform, consider Covidence or DistillerSR, which manage screening, data extraction, and quality assessment [12]. If you need a free, collaborative screener, Rayyan is an excellent starting point [9]. For reviews incorporating AI-based prioritization, explore tools like EPPI-Reviewer or SWIFT-Review [12]. Always start with a pilot test on a small subset of your references.

• Problem: My team and I are self-taught on an automation tool, and we lack confidence. This is a common barrier to adoption [9]. Where can we find reliable training?

  • Solution: First, consult the official knowledge base and tutorial videos provided by the tool developer. Second, seek out method papers that validate the tool's use (see Table 1). Third, engage with the research community through forums or networks like the International Collaboration for the Automation of Systematic Reviews (ICASR).

Workflow and Process Efficiency

• Problem: The promised time savings from automation aren't materializing. Our screening phase is still taking too long.

  • Solution: This often stems from a poorly configured workflow. Ensure you are using the tool's deduplication functions before screening begins. For AI-powered tools, the "work saved" is not immediate; you must first manually screen a sufficient batch of references (often 500-1000) to train the algorithm before it can effectively prioritize the remainder [10]. Re-evaluate your inclusion/exclusion criteria for clarity.

• Problem: We have discrepancies in how different reviewers apply labels during screening, undermining the AI model's learning.

  • Solution: Before full screening, conduct a calibration exercise. All reviewers should independently screen the same 50-100 abstracts, compare decisions, and discuss disagreements to refine the protocol. Use your tool's feature to resolve conflicts formally. This establishes consistency, which is crucial for both manual and automated screening accuracy [12].

Technical Performance and Accuracy

• Problem: Our AI-based screening tool is excluding too many relevant studies (high false negatives). How do we improve recall?

  • Solution: High recall (minimizing false negatives) is paramount [10]. First, check if your training set is representative and large enough. Second, adjust the tool's confidence threshold to be more inclusive. Third, consider a hybrid approach. A study found that a rules-based filter looking for the co-occurrence of key study characteristics (like Exposure and Outcome) in an abstract achieved 98% recall, potentially outperforming some standard ML classifiers [10]. You can apply such a filter before or after AI screening as a safety check.

• Problem: The tool's performance seems erratic and different from validation studies we've read.

  • Solution: Remember that tool performance is question- and corpus-dependent. An AI model validated on clinical trial abstracts may not perform as well on ecotoxicological observational studies due to different writing conventions and terminology. Treat validation study metrics as a guide, not a guarantee. Continuously monitor your project's specific performance metrics.

Data and Collaboration

• Problem: We need to transfer data (e.g., screened references) from one tool to another, but we're worried about losing information.

  • Solution: Always export and back up your data at each major stage. Most tools allow export in RIS or CSV formats. When transferring, map fields carefully and perform a spot-check on a sample of records post-import to ensure fidelity. Document all data handling steps for reproducibility [12].

• Problem: Collaboration features in our tool are clunky, causing version control issues and communication gaps within the team.

  • Solution: Clearly define and document your collaboration protocol. Assign roles (e.g., who can make final inclusion decisions). Use the tool's built-in commenting or conflict resolution modules for all discussions related to specific studies to maintain an audit trail. Schedule regular sync-ups outside the tool to discuss process issues.

The table below summarizes key features of major automation tools, based on survey data and technical evaluations [9] [12] [10].

Table 1: Comparison of Major Systematic Review Automation Tools

Tool Name	Primary Screening Methodology	Key Features & Integration	Reported User Adoption & Notes
Covidence	Manual screening with AI prioritization (in some versions)	Manages title/abstract screening, full-text review, risk of bias (RoB), data extraction. Integrates with reference managers.	Top-used tool (45% of respondents). Commonly abandoned (15%), indicating potential usability challenges [9].
Rayyan	Manual screening with ML-based ranking and deduplication	Free, collaborative web app for blinding and resolving conflicts during screening.	Used by 22% of respondents; also highly abandoned (19%), suggesting users may outgrow its initial features [9].
DistillerSR	Configurable manual screening with AI assist	Highly customizable forms for screening and data extraction, strong compliance and audit trail features.	Robust platform for large-scale reviews; noted as abandoned by 14% of users [9].
EPPI-Reviewer	Manual screening with active learning (AI prioritization)	Supports complex review types (e.g., meta-narrative, framework synthesis). Code is open-source.	Part of the "Big Four" comprehensive platforms. Known for active learning capabilities [12].
JBI SUMARI	Manual screening	Supports systematic reviews, umbrella reviews, and scoping reviews across diverse fields.	Developed by the Joanna Briggs Institute; part of the comprehensive platform suite [12].
PECO/EO Rule-Based Filter [10]	Automated exclusion based on missing key characteristics	Uses NLP to detect if Exposure and Outcome terms are absent from an abstract.	Not a standalone tool, but a method. Research demonstrated 93.7% exclusion rate with 98% recall, offering a high-recall pre-screening filter [10].

Experimental Protocol: PECO-Based Automated Screening

The following protocol details a validated, rules-based methodology for automating the initial screening of observational studies in fields like ecotoxicology. This approach can be implemented using text-mining software (e.g., the General Architecture for Text Engineering - GATE) or as a pre-processing step before using commercial screening tools [10].

Detailed Methodology [10]:

• Search & Corpus Creation: Execute your systematic search strategy in relevant databases (e.g., PubMed, Web of Science, Environment Complete). Import results into a reference manager, remove duplicates, and export the titles and abstracts of all unique references into a plain text format suitable for text mining.

• Development of Characteristic Dictionaries: For your specific review question, create controlled vocabularies.

  • Population (P): Terms describing the organisms or systems studied (e.g., "Daphnia magna," "zebrafish embryo," "soil microbiome").
  • Exposure (E): Terms for the chemical or stressor (e.g., "microplastic," "herbicide," "heavy metal," "temperature stress").
  • Outcome (O): Terms for the measured effects (e.g., "mortality," "growth inhibition," "gene expression," "reproductive success").
  • Confounders (C): (Optional) Terms for adjusting factors. This study found confounder terms were rarely stated in abstracts and their inclusion reduced screening efficiency [10].

• Text Mining and Rule Execution: Using a text-mining platform (e.g., GATE), implement a rule-based algorithm. The algorithm parses each abstract sentence, identifies key nouns and phrases, and matches them against the P, E, C, O dictionaries. The output is a simple binary code for each abstract indicating the presence or absence of phrases from each category.

• Application of Screening Threshold: Apply a pre-defined inclusion rule. The validation study found the most effective rule was: "Include a study for manual screening only if the algorithm detects terms for both Exposure (E) AND Outcome (O) in the abstract." Studies missing either E or O terms are automatically excluded. This rule achieved a recall of 98%, meaning it missed only 2% of truly relevant studies, while saving approximately 90% of the manual screening workload [10].

• Validation and Manual Review: The final step is to manually screen the subset of studies flagged as "includes" by the algorithm. It is critical to document the performance of the automated step (calculating its recall and precision against a small, manually screened sample) in your systematic review methods section.

The Researcher's Toolkit: Essential Materials & Reagents

Table 2: Key Research Reagent Solutions for Automated Screening Experiments

Item	Function in the Experimental Protocol	Notes & Considerations
Reference Corpus	The primary "reagent": A cleaned, deduplicated set of study titles and abstracts in machine-readable format (e.g., XML, JSON, plain text).	Quality is critical. Ensure abstracts are correctly matched to citations. Missing abstracts will be auto-excluded, potentially lowering recall.
Characteristic Dictionaries	Controlled vocabularies defining key concepts (P, E, O) for the NLP algorithm. Act as specific "detection probes."	Must be developed with domain expertise. Start from MeSH terms or authoritative glossaries. Requires iterative refinement and testing.
Text-Mining Software (e.g., GATE)	The "instrument" for executing the rule-based screening protocol. Processes the corpus using the dictionaries and linguistic rules.	GATE is open-source and provides a framework for developing custom processing pipelines. Alternatively, scripts can be written in Python (using NLTK, spaCy) or R.
Gold Standard Test Set	A subset of references (min. 50-100) that have been definitively classified (include/exclude) by human experts.	Used to calibrate dictionaries and validate the algorithm's performance (calculate recall/precision). Essential for reporting methodology.
Deduplication Tool	A pre-processing tool to remove duplicate records from multiple database searches.	Built into many reference managers (EndNote, Zotero) and systematic review platforms (Covidence, Rayyan). Critical for an accurate workflow.
Reporting Checklist (PRISMA)	A guideline framework for transparently reporting the entire review process, including the use of automation tools.	Using automation affects the PRISMA flow diagram. You must report the number of records excluded by the automation tool and its performance [12].

For researchers conducting systematic reviews in ecotoxicology, the Ecotoxicology (ECOTOX) Knowledgebase is an indispensable, publicly available resource for streamlining the initial evidence-gathering phase [6]. It is a comprehensive, curated database that provides information on the adverse effects of single chemical stressors on ecologically relevant aquatic and terrestrial species [6]. By compiling peer-reviewed test results into a structured, searchable format, ECOTOX addresses one of the most time-consuming steps in systematic reviews: the identification and collation of relevant toxicity data.

The database is curated from over 53,000 scientific references, encompassing more than one million test records for over 13,000 species and 12,000 chemicals [6]. This vast repository allows researchers to rapidly access toxicity benchmarks, inform ecological risk assessments, and support chemical registration processes without starting literature searches from scratch [6]. Within the context of automating systematic review screening, tools like ECOTOX serve as a critical pre-filtered data layer. They reduce the volume of primary literature that must be manually screened by sophisticated AI-driven tools (e.g., SWIFT-Active Screener, EPPI-Reviewer) in later stages, thereby accelerating the entire evidence synthesis workflow [12] [13].

The following table summarizes the core attributes and relevance of the ECOTOX Knowledgebase to automated systematic reviewing:

Table: The ECOTOX Knowledgebase as a Foundational Resource for Automated Screening

Attribute	Description	Role in Systematic Review Automation
Data Scope	>1M test records; 13K species; 12K chemicals; from 53K references [6].	Provides a massive, pre-identified corpus of relevant studies, reducing initial search burden.
Source Quality	Data abstracted from peer-reviewed literature via exhaustive search protocols [6].	Ensures data quality and reliability for the downstream review process.
Key Functionality	Search by chemical, species, or effect; advanced filtering; data visualization [6].	Enables rapid, targeted queries to gather a precise subset of data for a review question.
Regulatory Utility	Used to develop water quality criteria, ecological risk assessments, and support TSCA evaluations [6].	Directly supports regulatory-focused systematic reviews common in ecotoxicology.
Integration Potential	Data can be exported for use in other screening and analysis tools [6].	Serves as a high-quality data feed for dedicated systematic review software platforms.

Core Workflow: Integrating Curated Databases with Active Learning Tools

The most efficient modern systematic reviews in ecotoxicology combine the breadth of curated databases with the intelligent prioritization of active learning screening tools. This integration creates a hybrid workflow that significantly enhances efficiency.

The foundational step involves using the ECOTOX Knowledgebase to execute a precise, high-recall query based on the review's PICO criteria (Population/Plant, Intervention/Chemical, Comparator, Outcome) [6] [14]. The resulting set of literature citations and associated test records forms the initial corpus. This corpus is then imported into an active learning systematic review platform like SWIFT-Active Screener or EPPI-Reviewer [12] [13]. These platforms use machine learning models that learn from a reviewer's initial inclusion/exclusion decisions. They subsequently prioritize the remaining unscreened documents, pushing the most likely-to-be-relevant articles to the top of the queue [13]. This allows reviewers to identify the majority of relevant articles after screening only a fraction of the total list, achieving significant time savings [13].

Diagram: Integrated Workflow for Semi-Automated Evidence Gathering. This process combines the targeted data retrieval of curated databases with the intelligent prioritization of active learning tools to streamline screening [6] [13].

Technical Support Center: Troubleshooting Guides & FAQs

This section addresses common technical and methodological challenges researchers face when using curated databases and automation tools for systematic reviews.

FAQ 1: Data Retrieval and Handling

Q1: My query in the ECOTOX Knowledgebase returned an overwhelming number of results. How can I refine it to be more manageable for screening? A: An overly broad result set undermines efficiency. Use ECOTOX's 19 available filter parameters strategically [6]. Start by applying filters for the most critical aspects of your review protocol:

• Effect/Endpoint: Filter for the specific toxicity endpoints (e.g., "LC50," "mortality," "reproduction").
• Test Duration: Differentiate between acute and chronic studies.
• Exposure Medium: Specify aquatic (freshwater/saltwater) or terrestrial.
• Species Taxonomic Group: Filter to your relevant groups (e.g., "fish," "aquatic invertebrate"). Export the refined list and use it as the primary corpus for your screening tool. This targeted approach reduces noise and improves the performance of subsequent active learning algorithms [14] [13].

Q2: How do I handle the export from ECOTOX to ensure compatibility with my systematic review software (e.g., Covidence, SWIFT-Active Screener)? A: Compatibility is key for a smooth workflow. ECOTOX allows you to customize output selections from over 100 data fields during export [6]. For a seamless import into most screening tools:

• Ensure you export the core citation metadata (Author, Title, Journal, Year, DOI/PMID) as a standard format like .csv or .ris.
• The ECOTOX-specific test data (species, chemical, effect values) can be exported in a separate file. This detailed data is crucial for the subsequent data extraction phase after screening.
• Consult the "Help" section of your chosen systematic review software for specific import formatting requirements. Most modern tools accept standard bibliographic formats [12] [13].

FAQ 2: Integration with Screening Automation

Q3: The active learning model in my screening tool doesn't seem to be prioritizing relevant articles accurately. What could be wrong? A: Poor model performance often stems from an inadequate or biased initial "seed" set. The active learning model relies on your initial screening decisions to learn [13]. To fix this:

• Screen a Larger, Random Seed Set: Before relying on prioritization, screen a randomly selected batch of 100-200 articles. This gives the model a more representative foundation.
• Ensure Consistent Application of Inclusion Criteria: Review your protocol. Inconsistency in early decisions confuses the model. Dual screening on the seed set can improve consistency.
• Check Corpus Quality: If your initial corpus from ECOTOX is still too broad or off-topic, the model will struggle. Return to ECOTOX and refine your search with additional filters [6] [13].

Q4: How do I know when to stop screening with an active learning tool? When is it safe to assume I've found all relevant articles? A: You should not stop screening simply because relevant articles stop appearing consecutively. Reliable active learning tools like SWIFT-Active Screener incorporate a statistical recall estimation model [13]. This model continuously estimates the number of relevant articles remaining in the unscreened pile. A common best practice is to set a stopping threshold, such as screening until the model estimates with high confidence that over 95% of all relevant articles have been found. This provides a objective, data-driven stopping point instead of an arbitrary one [13].

FAQ 3: Regulatory and Validation Context

Q5: How can I validate that my semi-automated review process using these tools is robust enough for regulatory submission (e.g., for REACH, TSCA)? A: Regulatory acceptance hinges on transparency and methodological rigor. Your review protocol must pre-specify the use of these tools. Key steps include:

• Documenting Search & Filtering: Precisely record all ECOTOX search terms and filters used [6].
• Describing the Automation Method: Detail the active learning tool used, the seed set size, and the stopping criterion (e.g., 95% estimated recall) [13].
• Performing Quality Checks: Even with automation, a human quality check is essential. Plan to double-screen a random sample (e.g., 10-20%) of the excluded records to validate the model's accuracy and ensure no relevant studies were missed. This quality assurance data should be reported [12].

Q6: Are there other key EPA tools that complement ECOTOX in the evidence gathering and review process? A: Yes, the EPA's CompTox suite offers complementary tools. A critical one is the EPI Suite, a screening-level tool that estimates physical/chemical properties and environmental fate [15]. While ECOTOX provides observed toxicity data, EPI Suite's ECOSAR module can predict aquatic toxicity for chemicals with little or no available experimental data using Structure-Activity Relationships (SARs) [15]. This can be useful for prioritizing chemicals for review or filling data gaps. However, per EPA guidance, EPI Suite estimates "should not be used if acceptable measured values are available" [15].

Successful automation of systematic reviews requires a combination of specialized digital tools and a clear understanding of the experimental data being synthesized. The table below outlines key resources.

Table: Research Reagent Solutions: Digital Tools & Experimental Data Components

Tool / Resource Name	Type	Primary Function in Review Automation	Key Consideration for Ecotoxicology
ECOTOX Knowledgebase [6]	Curated Database	Provides pre-identified, structured toxicity data as a high-quality starting corpus for screening.	Contains ecologically relevant species data. Must be queried carefully to align with review PICO.
SWIFT-Active Screener [13]	Active Learning Screening Software	Uses machine learning to prioritize references during title/abstract screening, drastically reducing workload.	Effective performance depends on a well-defined initial corpus (e.g., from ECOTOX).
EPPI-Reviewer, Covidence, DistillerSR [12]	Comprehensive Systematic Review Platform	Manages the entire review pipeline (screening, data extraction, risk of bias) in a collaborative, online environment.	Ensure the platform's data extraction forms can capture ecotoxicology-specific fields (e.g., test species, endpoint, exposure regime).
EPA EPI Suite (ECOSAR) [15]	Predictive (QSAR) Tool	Provides predicted ecotoxicity values for data-poor chemicals, aiding in prioritization or gap analysis.	A screening-level tool only. Predictions must be clearly distinguished from experimental data in the review.
Toxicity Test Data (from primary studies)	Experimental Evidence	The fundamental material for synthesis. Includes details on species, chemical, concentration, duration, endpoint, and measured effect.	Critical to extract all relevant metadata (e.g., OECD test guideline, water chemistry) for use in sensitivity and bias analyses.
Digital Object Identifier (DOI)	Reference Identifier	Enables reliable linking between curated database records, screening tool imports, and full-text documents.	Verifying DOIs during the initial data export/import phase prevents matching errors later.

Experimental Protocol: Validating an Automated Screening Workflow

To empirically assess the efficiency gain from integrating a curated database with an active learning screener, researchers can follow this validation protocol.

Title: Protocol for Benchmarking a Semi-Automated Screening Workflow in Ecotoxicology Systematic Reviews. Objective: To compare the screening efficiency and recall accuracy of a traditional screening approach versus a hybrid (ECOTOX + Active Learning) approach for a defined review question. Materials: Access to the ECOTOX Knowledgebase [6], a licensed active learning screening tool (e.g., SWIFT-Active Screener [13]), a standard reference management tool. Method:

• Define a Test Review Question: Select a focused question (e.g., "What are the chronic toxicity values of Chemical X to freshwater invertebrates?").
• Create a Gold Standard Reference Set: Manually perform an exhaustive, traditional systematic search across multiple bibliographic databases (e.g., PubMed, Scopus, Web of Science) for the test question. Have two independent reviewers screen all retrieved records to establish a final "gold standard" set of included studies.
• Execute the Hybrid Workflow:
  • Query the ECOTOX Knowledgebase using the test question's key terms and apply relevant filters [6].
  • Export the results and import them into the active learning screening tool.
  • Follow the tool's active learning process, using its recommended stopping criterion (e.g., 95% recall estimation) [13].
• Benchmarking Analysis:
  • Primary Outcome (Efficiency): Record the total number of records screened in the hybrid workflow before stopping. Calculate the percentage reduction in screening effort compared to the total number screened in the traditional method.
  • Primary Outcome (Accuracy): Compare the final set of included studies from the hybrid workflow against the "gold standard" set. Calculate recall (percentage of gold standard studies found) and precision (percentage of included studies that are relevant).
• Validation: A valid hybrid workflow should achieve recall ≥ 95% while demonstrating a screening workload reduction of ≥ 50% compared to the traditional approach.

This protocol provides a framework for researchers to validate their own automated processes, ensuring they are both efficient and trustworthy for informing regulatory decisions and ecological risk assessments [16] [17].

Technical Support Center: HTS & Computational Toxicology Platform

Frequently Asked Questions (FAQs)

Q1: What are the most common causes of high false-positive rates in my high-throughput screening (HTS) assay for endocrine disruption? A: High false-positive rates in endocrine HTS (e.g., ER/AR transactivation assays) are frequently due to: 1) Compound interference (auto-fluorescence, quenching), 2) Cytotoxicity at test concentrations masking specific activity, 3) Non-specific binding to assay components, and 4) Edge effects in microplates due to evaporation. Implement counter-screens (viability assays) and use orthogonal assay confirmation.

Q2: How do I handle and process large, heterogeneous data streams from multiple HTS and high-content screening (HCS) platforms for systematic review? A: Utilize a structured data pipeline: 1) Ingestion: Use standardized formats (e.g., AnIML, ISA-TAB). 2) Normalization: Apply plate-based controls (Z', Z-factor) and robust statistical normalization (B-score). 3) Integration: Employ a centralized database with ontology-based tagging (e.g., ECOTOX, ChEBI). Automation tools like SWIFT-Review or ASReview can then be applied to the curated dataset for screening prioritization.

Q3: Why is my concentration-response curve fitting unstable when deriving AC50 values for ToxCast/Tox21 data? A: Unstable fits often stem from: 1) Insufficient data points across the critical effect range, 2) High variability in replicate measurements, 3) Inappropriate model selection (e.g., using Hill model for non-monotonic data). Ensure at least 10 concentrations with triplicate reads, and use suite-fitting algorithms (like those in the R tcpl package) that test multiple models and flag ambiguous fits.

Q4: What are the key validation steps when applying a machine learning model to predict in vivo toxicity from in vitro HTS data? A: Critical steps include: 1) External Validation: Testing on a wholly independent compound set not used in training. 2) Applicability Domain Assessment: Defining the chemical space where predictions are reliable. 3) Performance Metrics: Reporting AUC-ROC, precision-recall, and confusion matrices. 4) Mechanistic Plausibility: Ensuring predictions align with known adverse outcome pathways (AOPs).

Troubleshooting Guides

Issue: Low Assay Robustness (Z' < 0.5) in a Cell-Based Viability HTS.

• Check 1: Confirm cell viability and passage number at time of seeding. Use cells within passage 5-20.
• Check 2: Verify liquid handler precision for nanoliter dispensing. Perform dye-based dispense verification.
• Check 3: Monitor incubation conditions (CO2, temperature, humidity) for consistency.
• Check 4: Re-optimize positive/negative control concentrations. A shallow control response window lowers Z'.

Issue: Inconsistent Readout from a High-Content Imaging Cytotoxicity Assay.

• Step 1: Check for fluorescence crosstalk between channels. Include single-stain controls and adjust emission filters.
• Step 2: Validate focus stability across the plate. Use autofocus offset maps or whole-well focusing algorithms.
• Step 3: Standardize image analysis pipeline. Use supervised machine learning for segmentation (e.g., CellProfiler Analyst) to adapt to morphological changes induced by compounds.

Issue: Failure in Automated Data Extraction for Systematic Review Screening.

• Step 1: Audit the source PDFs. Poor OCR quality is the primary cause. Use pre-processors to enhance PDF image quality or seek native text versions.
• Step 2: Refine your natural language processing (NLP) query. Overly broad terms increase false positives; too specific increases false negatives. Iterate on a test set.
• Step 3: Implement active learning feedback. Tools like RobotAnalyst or Abstractx allow user feedback on relevance, which retrains the classifier in real-time to improve subsequent screening.

Table 1: Performance Metrics of Common HTS Assays in Tox21 Portfolio

Assay Target (PubChem AID)	Avg. Z'-Factor	Signal-to-Noise Ratio	False Positive Rate (%)	False Negative Rate (%)
NRf2 Response (743077)	0.72	12.5	4.2	7.8
p53 Activation (743079)	0.65	8.2	6.1	9.5
Mitochondrial Tox (743122)	0.58	6.8	8.5	12.3

Table 2: Comparison of Automation Tools for Systematic Review Screening

Tool Name (Version)	Recall (%)	Precision (%)	Workload Savings (%)	Supported File Formats
SWIFT-Review (v2.0)	98.5	35.7	~70	PDF, TXT, MEDLINE, RIS
ASReview (v1.0)	99.1	30.2	~90	CSV, RIS, TSV, Excel
RobotAnalyst (v1.0)	96.8	42.1	~75	PDF, PubMed IDs
DistillerSR (Enterprise)	95.0*	50.0*	~60*	All major formats

*Values based on published case studies; tool uses both NLP and manual rules.

Detailed Experimental Protocols

Protocol 1: HTS Assay for Cytotoxicity (ATP Content)

• Plate Seeding: Seed HEK293 or HepG2 cells in 1,536-well plates at 1,000 cells/well in 5 µL growth medium. Incubate (37°C, 5% CO2) for 24 h.
• Compound Addition: Using a pintool or acoustic dispenser, transfer 23 nL of test compound from a 10 mM DMSO stock. Include controls: DMSO only (negative), 100 µM Staurosporine (positive cytotoxic).
• Incubation: Incubate compound with cells for 48 hours.
• ATP Detection: Add 3 µL of CellTiter-Glo 2.0 reagent. Shake orbitally for 2 min, incubate at RT for 10 min to stabilize luminescent signal.
• Readout: Measure luminescence on a plate reader (integration time: 0.5-1 sec/well).
• Analysis: Normalize raw luminescence: % Viability = (RLUsample - RLUpositive) / (RLUnegative - RLUpositive) * 100. Calculate Z' = 1 - [3*(SDpositive + SDnegative) / |Meanpositive - Meannegative|].

Protocol 2: Building an Active Learning Model for Abstract Screening

• Data Preparation: Compile a corpus of title-abstract records from a PubMed/Web of Science search. Manually label a seed set (e.g., 100 records) as "relevant" or "irrelevant."
• Feature Extraction: Convert text to TF-IDF (Term Frequency-Inverse Document Frequency) vectors using n-grams (1-2 words).
• Model Initialization: Use a naive Bayes or SVM classifier within an active learning framework (e.g., ASReview software).
• Iterative Screening: The model ranks remaining abstracts by relevance probability. Reviewer screens the top 10-20 records, providing new labels.
• Model Update: The classifier retrains after each batch of labeled records.
• Stopping Criterion: Continue until a pre-defined threshold is met (e.g., 100 consecutive irrelevant abstracts).

The Scientist's Toolkit: Research Reagent Solutions

Item (Supplier Example)	Function in HTS/Toxicology
CellTiter-Glo 2.0 (Promega)	Luminescent ATP quantitation for viability/cytotoxicity.
Beta-lactamase Reporter Gene Cell Lines (Thermo Fisher)	Engineered cells for nuclear receptor screening (Tox21).
HuMo-DC (Hµrel)	Human primary cell co-culture for immunotoxicity screening.
UPLC-MS/MS System (Waters, Agilent)	Quantitative analytical chemistry for exposure assessment.
1,536-Well Microplates (Corning)	Ultra-high-throughput assay format.
Echo 650 Acoustic Dispenser (Labcyte)	Contactless, precise transfer of compounds/DMSO.
CellProfiler (Broad Institute)	Open-source HCS image analysis software.
tcpl R Package (US EPA)	Curve-fitting and data analysis pipeline for ToxCast data.

Workflow and Pathway Diagrams

From Setup to Synthesis: A Practical Workflow for Automated Screening

Within the demanding landscape of ecotoxicology research, the synthesis of evidence through systematic reviews is paramount for chemical safety assessment and regulatory decision-making. However, the traditional process is labor-intensive, often involving the manual screening of thousands of studies [18]. This article establishes a foundational technical support center, framed within a thesis on automation tools, to empower researchers in developing robust protocols and formulating precise eligibility criteria—the critical first steps toward implementing efficient, automated screening workflows.

Technical Support Center: Troubleshooting Guides & FAQs

This section addresses common challenges researchers encounter when initiating a systematic review with an eye toward automation.

Q1: How do I formulate a precise research question and eligibility criteria suitable for automation?

• Answer: Use a structured framework. The PICOST (Population, Intervention, Comparators, Outcome, Study design, Time period) framework is recommended for formulating the review question [19]. In ecotoxicology, this often adapts to PECO (Population, Exposure, Comparator, Outcome) [18]. Eligibility criteria must directly extend from these components. For automation, clarity is non-negotiable. Ambiguous criteria will confuse both human screeners and machine learning algorithms. Precisely define each element (e.g., "fathead minnow (Pimephales promelas) larvae" not just "fish"; "measured concentration of Bisphenol-A" not just "BPA exposure").

Q2: What are the key elements of a review protocol, and why is it critical for automated screening?

• Answer: A protocol is an a priori plan that minimizes bias and ensures reproducibility. Key elements include: administrative details; background and rationale; clearly defined PICOST/PECO question; explicit eligibility criteria; detailed search strategy for all databases; descriptions of the study selection process, data extraction, and risk-of-bias assessment methods; and a data synthesis plan [20]. For automation, the protocol is the source code. It defines the "rules" (eligibility criteria) that automated tools will help enforce and the workflow they will accelerate. It must be registered (e.g., PROSPERO) before screening begins [21].

Q3: Our initial search yields too many results. How can we refine it without compromising comprehensiveness?

• Answer: This is a common issue. First, ensure your eligibility criteria are sufficiently specific. Next, collaborate with a research librarian to troubleshoot the search strategy [21]. Use Boolean operators (AND, OR, NOT) effectively, but use NOT with caution to avoid accidentally excluding relevant studies [21]. Leverage "benchmark articles"—key papers you know should be included—to test your search string's sensitivity [21]. Finally, understand that high sensitivity is expected; a broad search retrieving many irrelevant articles is a prime use case for automation tools that can prioritize or exclude records.

Q4: What tools are available to assist with the screening phase, and how do they work?

• Answer: Several tools use machine learning to expedite screening. They typically require an initial set of human decisions ("training") and then predict the relevance of remaining records.
  • Rayyan: A free, widely-used web tool that facilitates collaborative screening and uses a relevancy classifier to prioritize citations [18] [22].
  • ASReview: An open-source tool that actively learns from your inclusion/exclusion decisions to present the most relevant records next [22].
  • SWIFT-Review: Provides interactive literature prioritization and categorization using text mining [22].
  • PICO Portal: A platform that assists with deduplication, screening, and highlighting PICO concepts in text [22].

Q5: We found an existing systematic review on a similar topic. Should we proceed?

• Answer: Conducting a duplicative review should be avoided [21]. First, thoroughly search for published reviews and protocols in registries like PROSPERO. If you find an overlapping review, assess its currency, scope, and quality. Your work is justified only if you are addressing a new question, updating outdated evidence, or using a significantly different methodology [21]. A librarian can help you navigate this decision [23].

Q6: How long does a systematic review typically take, and how does automation change this?

• Answer: A full systematic review typically takes 12 to 18 months [24]. The most time-consuming phase is screening titles and abstracts [18]. Automation tools do not perform the review for you but can significantly reduce this burden. Evidence suggests machine learning classifiers can reduce the number of abstracts needing manual screening by 30-70% [18], potentially saving weeks or months of work.

Performance Data & Experimental Protocols

Quantitative Comparison of Screening Approaches

The following table summarizes the performance of different screening methodologies, highlighting the efficacy of rule-based automation.

Table 1: Performance Comparison of Systematic Review Screening Methods

Screening Method	Core Principle	Typical Work Saved	Key Advantage	Primary Limitation
Manual Screening	Human review of all titles/abstracts	0% (Baseline)	High judgment capability; handles ambiguity.	Extremely time-consuming and labor-intensive [18].
ML-Powered Prioritization (e.g., Rayyan)	Ranks studies by relevance using word similarity [18].	Not fixed; accelerates finding includes.	Reduces time to first inclusion; good for early stopping.	Does not fully automate exclusion; final recall uncertain.
Rule-Based Automated Exclusion (PECO Detection) [18]	Excludes studies lacking predefined key characteristics.	Up to 93.7% (for EO rule)	High, quantifiable work reduction; transparent logic.	Dependent on quality of abstracts and extraction rules.
High-Throughput Ecotoxicology Paradigms [25]	Applies lab automation (e.g., fluidics, imaging) to in vitro/vivo bioassays.	N/A (Primary research)	Generates standardized, machine-readable toxicity data.	Not a screening tool for literature; generates new data for future reviews.

Detailed Experimental Protocol: Automated Screening via PECO Extraction

This protocol, based on published research [18], details steps to implement a rule-based automated screening module.

Objective: To automatically exclude studies from a systematic review search results that have a high probability of being irrelevant, based on the absence of key PECO (Population, Exposure, Comparator, Outcome) elements in their abstracts.

Materials & Software:

• Reference Set: Bibliographic records (title and abstract) from systematic review search, exported in XML or text format.
• Text Engineering Platform: General Architecture for Text Engineering (GATE) or similar natural language processing software [18].
• Custom Dictionaries & Rules: Domain-specific dictionaries for Exposure and Outcome terms relevant to the review (e.g., chemical names, ecological endpoints). Semantic rules to identify phrases describing Population and Comparator/Confounders.

Procedure:

• Training Set Creation: From the total references, identify a subset of included and excluded studies from a pilot manual screen (e.g., 20-50 studies) [18].
• Dictionary Development: Using the training set, manually curate comprehensive lists of keywords and phrases for Exposure and Outcome. For example, for a review on microplastics, exposure terms would include "microplastic," "nanoplastic," "polyethylene," etc.
• Rule Development in GATE: Create "JAPE" (Java Annotation Patterns Engine) transducers in GATE. These are grammatical rules that identify patterns indicative of PECO elements.
  • Example Rule for Exposure: Match a phrase where a term from the Exposure dictionary is preceded by verbs like "exposed to," "treated with," or measurements like "concentration of."
• Algorithm Validation: Run the extraction algorithm on the training set. Calculate precision (percentage of correctly identified phrases) and recall (percentage of all relevant phrases found). Iteratively refine dictionaries and rules until performance is acceptable (e.g., F-score >85%) [18].
• Application to Full Dataset: Process all retrieved abstracts through the validated GATE pipeline. The output is a structured annotation marking text spans for each PECO element.
• Apply Screening Threshold Rule: Implement a logical rule to tag studies for inclusion or exclusion. For example, the highly effective "EO rule": IF (Exposure term is detected) AND (Outcome term is detected) THEN tag for manual screening; ELSE tag for automated exclusion [18].
• Human Verification: Manually screen all studies tagged for inclusion by the algorithm, plus a random sample (e.g., 10%) of excluded studies to validate rule performance and calculate final recall/work saved.

Detailed Protocol: High-Throughput Phenotypic Profiling (Cell Painting) for Ecotoxicology

This protocol aligns with emerging automation in primary research, which generates data for future reviews [25] [26].

Objective: To perform high-content, automated screening of chemical toxicity using morphological profiling in non-human vertebrate cell lines.

Materials & Reagents:

• Cell Line: Fish or other environmentally relevant vertebrate cell line (e.g., RTgill-W1 from rainbow trout).
• Staining Reagents: Cell Painting cocktail: dyes for nuclei (Hoechst), cytoplasm (Concanavalin A or phalloidin), endoplasmic reticulum, Golgi apparatus, and mitochondria [26].
• Automated Equipment: Robotic liquid handler, automated cell culture incubator, high-content imaging microscope (e.g., ImageXpress).
• Software: Image analysis software (e.g., CellProfiler) for feature extraction.

Procedure:

• Cell Seeding & Treatment: Using a robotic liquid handler, seed cells into 384-well microplates. After adherence, treat cells with a concentration gradient of the test chemical(s) and appropriate controls (vehicle, positive cytotoxicant).
• Staining & Fixation: At exposure endpoint (e.g., 48h), automate the steps of fixation, permeabilization, and staining with the Cell Painting dye cocktail.
• Automated Imaging: Plates are automatically loaded into a high-content imager, which acquires high-resolution fluorescent images from multiple sites per well across all channels.
• Morphological Feature Extraction: Images are processed by CellProfiler. The software identifies individual cells and measures ~1,500 morphological features (size, shape, texture, intensity) for each organelle stain.
• Data Analysis & Bioactivity Profile: The multidimensional data is normalized and analyzed. Treatments causing significant morphological changes are identified, and clustering analysis groups chemicals with similar phenotypic "fingerprints," inferring potential mechanisms of action.

Diagram: Workflow for Automated Systematic Review Screening

Diagram: High-Throughput Ecotoxicology (HITEC) Paradigm

The Scientist's Toolkit: Research Reagent Solutions

Table 2: Essential Tools for Automated Screening & High-Throughput Ecotoxicology

Item Category	Specific Tool/Reagent	Function in Protocol Development & Automation
Protocol & Project Management	PRISMA Checklist [19] [24]	Provides evidence-based minimum reporting items for protocols and reviews, ensuring completeness and transparency.
Eligibility Criteria Framework	PICOST / PECO Template [18] [19]	Provides a structured framework to define the research question and operationalize eligibility criteria for both humans and algorithms.
Text Mining & NLP Engine	General Architecture for Text Engineering (GATE) [18]	An open-source platform for building custom text processing pipelines to extract PECO and other key concepts from abstracts.
Machine Learning Screening	ASReview / Rayyan [22]	Open-source and free-to-use software that implements active learning to prioritize screening queues, reducing manual workload.
High-Throughput Bioassay	Cell Painting Assay Cocktail [26]	A multiplexed fluorescent dye set that labels multiple organelles, enabling high-content morphological profiling for chemical bioactivity screening.
Automated Imaging & Analysis	High-Content Imager & CellProfiler [25] [26]	Hardware and software for automated, quantitative capture and analysis of cellular phenotype images, generating rich datasets for toxicity prediction.
Reference Management	EndNote, Rayyan [18] [22]	Tools for deduplicating search results, managing citations, and facilitating collaborative screening among review team members.

Technical Support Center: Troubleshooting for Automated Screening Tools

This support center addresses common issues encountered when implementing AI-powered tools for automating the title/abstract screening phase of systematic reviews in ecotoxicology. Effective tool selection hinges on matching software capabilities to your project's specific scale, team structure, and review complexity.

Frequently Asked Questions (FAQs)

Q1: The AI model in our screening software (e.g., ASReview, Rayyan AI) is performing poorly, consistently prioritizing irrelevant studies. What steps should we take? A: Poor AI performance often stems from insufficient or biased initial training data. Follow this protocol:

• Pause live screening. Revert to the software's "exploration" or "training" mode.
• Re-evaluate your seed set. Manually screen a random sample of 50-100 records. Label them as "relevant" or "irrelevant" with high confidence.
• Ensure balance. If your topic is niche, aim for at least 15-20 relevant studies in the seed set. The AI needs clear positive examples.
• Re-train the model. Use this curated, balanced seed set to initiate or re-train the AI model. The software should now have a better foundational understanding of your inclusion criteria.
• Resume screening. Monitor the "relevancy ranking" as you screen; relevant studies should cluster toward the top.

Q2: Our multi-reviewer team is experiencing conflicts and inconsistencies in labeled records when using collaborative screening platforms. How can we resolve this? A: This is a workflow and calibration issue, not solely a software bug.

• Implement a pre-screening calibration exercise. Before the main screening, all reviewers independently screen the same pilot set of 50-100 articles using the shared platform.
• Calculate Inter-Rater Reliability (IRR). Use the platform's reporting tool or export data to calculate Cohen's Kappa or Percent Agreement.
• Hold a conflict resolution meeting. Discuss disagreements on the pilot set to clarify and refine the inclusion/exclusion criteria.
• Configure software settings. Enable "conflict resolution" workflows where disputed records are flagged for a third reviewer or lead investigator to make a final judgment. Ensure all users are assigned to the correct project phase.

Q3: We need to customize our screening workflow to include a specific data extraction field (e.g., "LOE: Level of Evidence") immediately after inclusion. How can we achieve this without breaking the workflow? A: Most advanced tools (e.g., DistillerSR, SysRev) allow for custom form creation.

• Access the study/form designer. Navigate to the project administration settings.
• Add a custom field. Create a new field with the label "LOE." Define the answer type (e.g., dropdown: "I, II, III, IV" or numeric).
• Apply conditional logic. Set the field to become mandatory only after the "Title/Abstract Screening" decision is set to "Include." This ensures screeners are not burdened with it prematurely.
• Test the workflow. Use a test project or a few dummy records to verify that the field appears and behaves as intended at the correct stage.

Comparative Performance Data of Common Screening Tools

Table 1: Feature Comparison of Selected Systematic Review Automation Tools Relevant to Ecotoxicology

Tool Name	Core AI/ML Capability	Collaboration Features	Customization Level	Ideal Project Scale
ASReview	Active Learning (Prioritization)	Limited (Basic sharing)	Low (Open-source; can modify code)	Small to Medium, single-reviewer focus
Rayyan	AI Suggestions & Deduplication	Strong (Multi-reviewer, blinding, conflict resolution)	Medium (Custom tags, filters)	Medium to Large, collaborative teams
DistillerSR	AI Rank & Relevance Scoring	Enterprise-grade (Complex roles, audit trails)	High (Custom forms, workflows, reporting)	Large, regulatory-compliant reviews
SysRev	AI Classifier & Prioritization	Strong (Dashboards, task assignment)	High (Custom data extraction forms)	Medium to Large, interdisciplinary teams

Experimental Protocol: Benchmarking AI Tool Performance

Objective: To empirically evaluate the workload savings offered by an AI-powered prioritization tool compared to traditional random screening for an ecotoxicology systematic review.

Methodology:

• Dataset Preparation: A validated benchmark dataset of approximately 5,000 citation abstracts from an existing ecotoxicology review is imported into the test software (e.g., ASReview). The "ground truth" inclusion/exclusion labels are hidden from the algorithm.
• Seed Set Creation: A random sample of 25 records (containing ~5 relevant studies) is selected to simulate the initial manual screening effort and used to train the AI model.
• Simulated Screening: The experiment runs in simulation mode. The AI model prioritizes the remaining records. The software records the cumulative number of relevant studies found (recall) versus the cumulative number of records screened.
• Control Arm: A separate simulation is run where records are presented in a random order.
• Outcome Measurement: The primary metric is Work Saved over Sampling at 95% recall (WSS@95). This calculates the percentage of records a reviewer does not have to screen before finding 95% of all relevant studies, compared to the random approach.

Example Results: In a simulation, the AI-prioritized order may achieve 95% recall after screening only 30% of the total dataset, whereas random order requires screening 95% of it. Therefore, WSS@95 = 95% - 30% = 65% workload reduction.

Title: AI-Powered Screening Simulation Protocol

The Scientist's Toolkit: Research Reagent Solutions for Automated Review

Table 2: Essential Digital "Reagents" for an Automated Screening Experiment

Item	Function in the Experiment	Example/Note
Benchmark Dataset	A pre-labeled collection of citations (relevant/irrelevant) used to validate and compare AI tool performance.	e.g., A publicly available systematic review dataset from the field of environmental toxicology.
Active Learning Algorithm	The core AI "engine" that queries the next most informative record to label, optimizing the discovery of relevant studies.	e.g., Support Vector Machines (SVM), Naïve Bayes, or neural networks embedded in tools like ASReview.
Deduplication Module	Identifies and merges duplicate citations from multiple databases (e.g., PubMed, Scopus, Web of Science) to prevent bias.	A critical pre-processing step in Rayyan, DistillerSR, and others.
Inter-Rater Reliability (IRR) Calculator	A statistical module (often built into collaboration tools) that quantifies screening consistency between reviewers (e.g., Cohen's Kappa).	Essential for ensuring protocol adherence in team-based screening.
PRISMA Flow Diagram Generator	A reporting tool that automatically populates the PRISMA flowchart based on screening decisions logged in the platform.	Saves significant time during the manuscript writing phase (feature in DistillerSR, SysRev).

Troubleshooting Guides & FAQs

Q1: After importing my references from EndNote, many records appear to be missing. What could be the cause? A: This is commonly due to duplicate records being automatically removed by the platform. Both Covidence and DistillerSR have strict deduplication protocols upon import. First, check the import report summary. If the issue persists, ensure your EndNote library exports all relevant fields (including abstracts) in a compatible format like RIS or PubMed XML. A preliminary deduplication in a reference manager before import can prevent unexpected record loss.

Q2: During title/abstract screening, the "Maybe" or "Conflict" pile is growing too large, slowing down progress. How can we refine our criteria? A: A large uncertain pile often indicates screening criteria that are too vague. Pause screening and conduct a "calibration exercise." Have all screeners independently review the same 50-100 records from the "Maybe" pile, then meet to discuss discrepancies. Use this discussion to clarify and explicitly rewrite inclusion/exclusion rules, adding specific examples. Update the platform's screening form with these new decision trees before proceeding.

Q3: We are experiencing significant lag or timeout errors when trying to screen references in Rayyan. What steps can we take? A: Rayyan's performance can degrade with very large review projects (>10k references) or when using many complex keywords/filters simultaneously. First, try clearing your browser cache or switching to a different browser (Chrome/Firefox are recommended). If the issue persists, break your project into smaller, manageable phases (e.g., screen by year of publication). For persistent issues with large datasets, consider platforms like Covidence or DistillerSR, which are engineered for higher-volume commercial research.

Q4: In DistillerSR, how do we handle a situation where a full-text document cannot be retrieved for a seemingly eligible study? A: DistillerSR has a built-in workflow for this. Log the item as "Awaiting Classification" and use the internal task assignment or comment system to delegate the retrieval effort. Document every retrieval attempt (e.g., library request, contact author, search in alternative repositories) directly in the study's history log. After exhausting all avenues (typically 3+ attempts), you can create a custom exclusion reason such as "Full text unavailable" to maintain an audit trail and ensure transparency in your PRISMA flow diagram.

Q5: During the full-text review stage in Covidence, a team member accidentally excluded a study that should have been included. Can this be reversed? A: Yes. A Covidence administrator for the review can reverse this. Navigate to the "Excluded" studies list, find the relevant study, and click "Return to screen." The study will be sent back to the previous stage (full-text review) for a new, independent decision. This action is logged. It is good practice to document the reason for the reversal in the study's notes to maintain protocol adherence.

Experimental Protocols for Key Cited Methodologies

Protocol 1: Implementing a Dual-Independent Blind Screening Workflow This protocol minimizes bias in the study selection process.

• Preparation: Two screeners are trained on the finalized, pilot-tested screening form with explicit criteria.
• Blinding: Using the platform's settings (e.g., in Rayyan: "Hide decisions"), screeners are blinded to each other's decisions.
• Parallel Screening: Each screener independently evaluates all records (title/abstract) against the criteria, marking them as "Include," "Exclude," or "Maybe."
• Conflict Identification: The platform's algorithm (e.g., Covidence's "Conflicts" tab) automatically flags records where decisions disagree.
• Consensus Meeting: Screeners meet to discuss only the conflicting records. They review the record and the criteria to reach a consensus decision.
• Adjudication: If consensus cannot be reached, a third reviewer (a senior researcher) adjudicates the final decision.

Protocol 2: Building and Testing a Complex Keyword Filter in DistillerSR This protocol uses DistillerSR's advanced AI and filtering tools to pre-sort references.

• Query Formulation: Define a set of keywords related to your PICO/PECO question (e.g., for an ecotoxicology review: "Daphnia magna," "chronic toxicity," "LC50").
• Filter Creation: In the DistillerSR "Designer" view, create a new "Word Wheel" or "AI-Assisted" filter. Input the keyword groups, using Boolean operators (AND, OR, NOT) to link them.
• Validation Test: Apply the filter to a known, gold-standard set of 50-100 references that you have pre-classified as relevant or irrelevant.
• Performance Metrics: Calculate the filter's sensitivity (proportion of truly relevant records it correctly identifies) and precision (proportion of records it flags that are truly relevant). See Table 1.
• Iterative Refinement: Adjust keywords and Boolean logic to maximize sensitivity (to avoid missing key studies) while maintaining acceptable precision. A final filter can be used to prioritize screening order.

Table 1: Platform Feature Comparison for Systematic Review Screening

Feature / Capability	Rayyan	Covidence	DistillerSR
Cost Model	Freemium (paid for advanced features)	Subscription per review	Enterprise Subscription
Deduplication	Basic	Advanced, configurable	Highly advanced, multi-method
Blind Screening	Yes	Yes	Yes
Conflict Resolution	Manual highlight	Dedicated conflicts tab & workflow	Configurable workflow automation
AI / ML Features	Keyword highlighting, semi-automatic deduplication	Priority screening (algorithmic sorting)	Advanced AI filters, continuous learning ranking
Export for PRISMA	Manual count extraction	Automated PRISMA flow diagram data	Fully automated PRISMA diagram generation
Ideal Project Size	Small to Medium (<5k references)	Medium to Large	Large, Complex, & Regulatory

Table 2: Performance Metrics from a Filter Validation Test (Hypothetical Data) Based on a test of 100 pre-classified references (70 relevant, 30 irrelevant).

Filter Version	Records Flagged	True Positives (TP)	False Positives (FP)	Sensitivity (TP/70)	Precision (TP/Flagged)
Initial Broad Filter	90	68	22	97.1%	75.6%
Refined Specific Filter	65	65	0	92.9%	100%

Workflow Visualizations

Dual-Phase Screening Workflow with Conflict Resolution

Decision Tree Logic for Screening a Single Study Record

The Scientist's Toolkit: Research Reagent Solutions for Screening

Item	Function in the Screening Process
Screening Protocol & Codebook	The foundational document defining the research question (PECO), explicit inclusion/exclusion criteria, and operational definitions for all variables. Serves as the "standard operating procedure" for all screeners.
Piloted Screening Form	The digital implementation of the codebook within the review platform (Covidence, DistillerSR, etc.). Must be piloted and refined before full use to ensure clarity and reduce screener disagreement.
Calibration Set of References	A small, pre-classified set of 20-50 references (both relevant and irrelevant) used to train and calibrate the screening team, ensuring consistent interpretation of the protocol.
Reference Management Library (e.g., EndNote, Zotero)	Used for initial collection, preliminary deduplication, and backup of records before import into the specialized screening platform.
Pre-defined Exclusion Reason Tags	A standardized list of exclusion reasons (e.g., "Wrong population," "Wrong exposure," "No control group") configured in the screening platform. Ensures consistent, analyzable data on why studies were excluded.

This technical support center addresses common challenges researchers face when implementing AI and Machine Learning (ML) tools to automate the screening phase of systematic reviews in ecotoxicology. The process involves using algorithms to prioritize, rank, and continuously learn from decisions made on thousands of research abstracts, significantly reducing manual workload.

Frequently Asked Questions (FAQs) & Troubleshooting Guides

Q1: Our initial model performs poorly, ranking irrelevant abstracts highly. What are the first steps to diagnose this? A1: This is often a training data issue.

• Check Your Seed Set: The "seed of known relevant" documents used to train the initial model must be of high quality and representative of your review's inclusion criteria. A small, non-representative seed set is a common point of failure.
• Actionable Protocol: Manually screen the first 100-200 abstracts from your corpus. Identify all relevant documents within this batch to create a robust, initial seed set of at least 20-30 relevant items. Retrain your model with this improved seed.
• Verify Data Format: Ensure your abstracts and labels are correctly formatted for your chosen tool (e.g., CSV columns for 'title', 'abstract', 'label').

Q2: How do we handle severe class imbalance (few relevant, many irrelevant abstracts) to prevent model bias? A2: Strategic sampling and algorithm choice are key.

• Protocol - Active Learning Loop: Use an Active Learning (AL) framework. The algorithm should strategically query you to label the most "uncertain" documents (those it cannot easily classify). This focuses human effort on informative samples, not random ones.
• Algorithm Settings: Ensure your tool uses algorithms designed for imbalance (e.g., SVM with balanced class weights, ensemble methods). Do not use simple random sampling for training.
• Evaluation Metric: Stop using accuracy. Use Recall (Sensitivity) as your primary metric to ensure you are capturing most of the relevant documents. Monitor precision to understand the trade-off.

Q3: What is the recommended workflow for integrating continuous learning, and why does model performance seem to degrade over time? A3: A structured workflow prevents degradation, often caused by concept drift.

• Detailed Protocol:
  • Initial Phase: Train model on the initial seed set.
  • Prioritization & Screening: Screen abstracts in the order of relevance rank provided by the model.
  • Batch Retraining: After every 50-100 new manual screenings, add the newly labeled data to the training set and retrain the model. Do not wait until the end.
  • Validation Checkpoint: After retraining, apply the model to a held-out validation set of known labels to monitor recall/precision. A significant drop may indicate drift.
  • Model Reset (if needed): If performance degrades, revert to the last stable model and consider retraining from scratch with all accumulated data, or adjust the algorithm's learning rate.

Q4: How do we quantitatively evaluate if the AI tool is saving time without missing critical studies? A4: Use the Work Saved over Sampling (WSS) metric at a specific recall level.

• Calculation Protocol:
  • Run your AI-assisted screening until you have identified 95% of the total relevant studies found in your manual process (Recall @95%).
  • Calculate the percentage of the total corpus you had to screen to reach this point.
  • WSS@95% = 100% - (% of corpus screened at Recall 95%).
  • A WSS@95% of 70% means you screened only 30% of the total abstracts to find 95% of relevant studies, saving 70% of the screening workload.

Q5: We are using a pre-trained NLP model (like BERT). Should we fine-tune it on our ecotoxicology corpus? A5: Yes, domain-specific fine-tuning is highly recommended.

• Fine-Tuning Protocol:
  • Gather Domain Text: Compile a large corpus of ecotoxicology literature (e.g., all abstracts from relevant journals).
  • Use Masked Language Modeling (MLM): Continue training the pre-trained model on this domain corpus to adapt its understanding of specialized vocabulary (e.g., "endocrine disruption," "LC50," "chronic toxicity").
  • Task-Specific Tuning: Subsequently, fine-tune this domain-adapted model on your specific labeled dataset of relevant/irrelevant abstracts for the classification/ranking task. This two-step process typically yields superior performance.

Data Presentation: Performance Metrics of AI-Assisted Screening

Table 1: Comparative Performance of Common Algorithms in Systematic Review Automation (Simulated Data based on recent literature).

Algorithm / Approach	Average Recall @95%	Average Work Saved (WSS@95%)	Key Strength	Consideration for Ecotoxicology
Naïve Bayes (Baseline)	91%	62%	Fast, simple, works with small data.	Lower precision; may struggle with complex terminology.
Support Vector Machine (SVM)	97%	75%	Effective in high-dimensional spaces.	Requires careful feature engineering and parameter tuning.
Random Forest	98%	78%	Robust to overfitting, handles non-linearity.	Less interpretable; can be computationally heavy.
Fine-Tuned BERT (or similar transformer)	99%	85%	Captures complex contextual language.	Requires significant computational resources for fine-tuning.

Experimental Protocols

Protocol 1: Building a Continuous Learning Active Learning System

• Data Preparation: Compile deduplicated abstract dataset. Annotate a seed set (min. 30 relevant, 70 irrelevant).
• Feature Extraction: Convert text to features using TF-IDF or pre-trained sentence embeddings (e.g., SciBERT).
• Model Initialization: Train a classification model (e.g., SVM) on the seed set.
• Active Learning Loop: a. The model scores all unlabeled abstracts. b. An "uncertainty sampling" query strategy (e.g., lowest prediction confidence) selects the n (e.g., 10) most uncertain abstracts for manual labeling. c. The human screener labels these n abstracts. d. These newly labeled abstracts are added to the training set. e. The model is retrained on the updated dataset. f. Repeat steps a-e until a stopping criterion is met (e.g., 100 consecutive irrelevant abstracts).
• Validation: Performance is tracked on a held-out validation set after each retraining cycle.

Protocol 2: Calculating Performance Metrics (WSS & Recall)

• Define Gold Standard: Manually screen the entire corpus (or a large random sample) to establish the complete set of relevant documents. This is your "truth set."
• Run AI Simulation: Simulate the AI-assisted process. Record the cumulative number of documents screened and the cumulative number of relevant documents found at each step, following the AI's ranking.
• Calculate Recall: At any point, Recall = (Relevant Found / Total Relevant in Truth Set).
• Identify Stopping Point: Determine the point in the ranked list where you achieved 95% recall.
• Calculate WSS@95%: WSS@95% = 1 - (Number of screened docs at 95% recall / Total docs in corpus).

Visualizations

AI Screening Workflow with Active Learning

Abstract Prioritization via Uncertainty Sampling

The Scientist's Toolkit: Research Reagent Solutions

Table 2: Essential Tools & Libraries for Automating Systematic Review Screening

Item / Solution	Category	Function / Purpose
ASReview	Open-Source Software Platform	An active learning-powered tool designed specifically for systematic review screening. Handles ranking, continuous learning, and evaluation out-of-the-box.
Rayyan	Web Application	A collaborative screening platform with basic ML prioritization features to expedite manual screening.
Python Scikit-learn	Machine Learning Library	Provides a wide array of algorithms (SVM, Naïve Bayes, Random Forest) and utilities for building custom text classification pipelines.
Transformers Library (Hugging Face)	NLP Library	Provides access to thousands of pre-trained language models (e.g., BioBERT, SciBERT, RoBERTa) for state-of-the-art text representation and classification.
PROBAST / AI-specific TRIPOD	Reporting Guideline	Tools to assess risk of bias and ensure transparent reporting of AI models used in research synthesis.
Zotero / EndNote	Reference Manager	Used to initially collect, deduplicate, and export citation data for processing in AI screening tools.
Custom Ecotoxicology Text Corpus	Training Data	A large collection of domain-specific text (abstracts, full texts) essential for fine-tuning generic language models to understand field-specific terminology.

Integrating Specialized Tools for Data Extraction and Management (e.g., SUMARI, EPPI-Reviewer)

Technical Support Center: Troubleshooting Guides & FAQs

This support center addresses common issues encountered when integrating tools like SUMARI and EPPI-Reviewer for automating systematic review screening in ecotoxicology.

Frequently Asked Questions (FAQs)

Q1: During the initial import of search results from databases (e.g., Scopus, PubMed) into EPPI-Reviewer, many records are duplicated. What is the primary cause and solution? A: The primary cause is importing results from multiple databases without first deduplicating using a consistent identifier (e.g., DOI). Use EPPI-Reviewer's built-in deduplication function before beginning screening. Navigate to References -> Check for duplicates. Select "DOI" as the primary matching field and "Title" as secondary. The software will identify clusters of potential duplicates for your review.

Q2: When using SUMARI for risk-of-bias assessment, the collaborative review feature is not updating in real-time for all team members. What steps should be taken? A: This is typically a project synchronization issue. First, ensure all users have the latest version of the SUMARI project file. The lead reviewer should: 1) Go to Project -> Sync History to check for conflicts. 2) Use Project -> Consolidate Reviews to merge all assessments. 3) Redistribute the consolidated project file. For persistent issues, use the manual backup/merge protocol detailed in the Diagram 2 workflow.

Q3: A critical error occurs during the automated priority screening process in EPPI-Reviewer's Classifier tool, halting the process. How do you diagnose and recover? A: First, check the Classifier job status under System Tasks. If it shows "Failed," note the error code. Common fixes include: 1) Insufficient Training: The classifier requires a minimum of 20+ inclusions. Ensure you have provided enough manually screened "included" studies. 2) Memory Error: For reviews >10k references, allocate more memory via the EPPI-Reviewer launcher settings. 3) Corpus Error: Reset the classifier (Classifier -> Advanced -> Reset current learning) and retrain.

Q4: Exported data tables from SUMARI for statistical analysis in R are missing crucial meta-data columns. How do you ensure a complete export? A: SUMARI uses a modular export system. You must export data from each module separately and merge them using a unique study ID. Do not rely on a single "complete" export. The key modules for ecotoxicology are: 1) Study Characteristics, 2) Risk of Bias, and 3) Outcome Data. Use the Export -> CSV function in each, and merge tables in your statistical software using the Study ID field.

Troubleshooting Guides

Issue: Failure in Automated Screening Workflow Integration

Symptoms: References screened in EPPI-Reviewer do not appear in the SUMARI risk-of-bias module, breaking the pipeline. Resolution Protocol:

• Verify Format: Ensure export from EPPI-Reviewer uses the correct format. Use Export -> References -> Select Tab-delimited (.txt) and include Review Inclusion Status.
• Check Mapping: In SUMARI, during import, explicitly map the EPPI-Reviewer inclusion column (e.g., Include) to SUMARI's screening status field.
• Use Bridge Script: If standard import fails, use the provided Python bridge script (see Table 2) to clean and convert the data.
• Log Audit: Check the activity.log file in both software directories for timestamped import errors.

Issue: Inconsistent Coding Schema Application Across Tools

Symptoms: Study characteristics (e.g., "test organism") coded in EPPI-Reviewer do not match the allowed values in SUMARI, causing import failures. Resolution Protocol:

• Pre-define Schema: Before screening, create a master coding schema (a controlled vocabulary) as a .csv file. See Table 1 for key fields.
• Import Schema First: Upload this schema into EPPI-Reviewer's coding tool and SUMARI's study characteristics module before starting.
• Validation: Use EPPI-Reviewer's reports function (Reports -> Coding Consistency) to check for deviations before exporting.
• Transform Data: Use the transformation rules in the bridge script to handle any minor syntactic differences (e.g., "Daphnia magna" -> "Daphnia magna").

Experimental Protocols

Protocol 1: Validating an Automated Screening Classifier in EPPI-Reviewer for Ecotoxicology Reviews

Objective: To train and validate a machine learning classifier to prioritize ecotoxicology records for manual screening. Methodology:

• Seed Training Set: Manually screen a random sample of at least 500 references from the total retrieved (e.g., 5,000-10,000). Label as "Include" or "Exclude."
• Classifier Training: In EPPI-Reviewer, navigate to Classifier -> New Classifier. Select the "Priority Screening" model. Specify the field containing your manual decisions as the target.
• Validation Design: Use the k-fold cross-validation option (k=10) within the tool. This partitions the training set to estimate performance.
• Performance Metrics: The tool outputs a table (see Table 1) including Recall (Sensitivity), Precision, and Work Saved over Sampling (WSS).
• Deployment: Apply the trained classifier to the unscreened references. The software will rank them from most to least likely to be relevant. Screen from the top until the desired recall level (e.g., 95%) is empirically confirmed.

Protocol 2: Exporting, Merging, and Analyzing Data Across SUMARI and EPPI-Reviewer

Objective: To create a unified dataset for meta-analysis from separate screening (EPPI-Reviewer) and data extraction (SUMARI) tools. Methodology:

• Export from EPPI-Reviewer: Export the final included study list with all coded characteristics: References -> Export -> Select Included studies only, format CSV.
• Export from SUMARI: From each module (Study Characteristics, Risk of Bias, Outcomes), export data as CSV.
• Data Merging Script: Execute a Python script (see Table 2) that:
  • Reads all CSV files.
  • Standardizes Study ID format (e.g., "Smith_2020").
  • Performs a left join, using the EPPI-Reviewer inclusion list as the primary key.
  • Outputs a single, tidy data file for statistical software.
• Quality Check: The script generates a summary log listing any studies with missing data from any module for manual follow-up.

Data Presentation

Table 1: Performance Metrics of a Classifier for Ecotoxicology Systematic Review Screening (Simulated Data)

Metric	Description	Target Value in Ecotoxicology	Example Result from Validation
Recall (Sensitivity)	Proportion of all relevant studies correctly identified by the classifier.	≥ 95% (to minimize misses)	98.2%
Precision	Proportion of classifier-predicted inclusions that are truly relevant.	Varies; higher reduces manual load.	45.5%
Work Saved over Sampling (WSS)	% of screening effort saved at a given recall level.	WSS@95% should be > 50%.	WSS@95% = 72.3%
Number of Training Studies	Manually screened studies used to train the model.	Minimum of 20-30 inclusions.	55 inclusions

Table 2: Essential Components for Data Integration Bridge Script

Component	Function	Example Tool/Library
Data I/O Handler	Reads/writes various file formats (CSV, RIS, TXT).	Python pandas library
Identifier Matcher	Aligns study records across tools using DOI, Title, Author/Year.	Fuzzy matching with thefuzz library
Schema Mapper	Translates coding values from one tool's schema to another's.	Custom dictionary/JSON mapping file
Log Generator	Creates an audit trail of merge decisions, conflicts, and errors.	Python logging module

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Digital Materials for Automated Screening Pipeline

Item	Function in Ecotoxicology Review Pipeline	Example/Specification
Reference Management File	Standardized container for search results from bibliographic databases.	RIS or ENW file format with DOI and Abstract fields.
Coding Schema File	Controlled vocabulary for key study characteristics (e.g., species, chemical, endpoint).	CSV file with columns: Field_Name, Allowed_Value, Definition.
Data Integration Script	Executable code to merge data from different specialized tools.	Python script using pandas (see Protocol 2).
Validation Test Set	A benchmark of pre-screened references to test classifier performance.	A .csv file of 100-200 references with known inclusion status, held back from training.

Visualizations

Title: Automated Systematic Review Screening and Data Extraction Workflow

Title: Troubleshooting Data Integration Failures Between Tools

Navigating Pitfalls and Maximizing Efficiency in Automated Workflows

Technical Support & Troubleshooting Center

This support center is designed for researchers conducting systematic reviews in ecotoxicology and related environmental sciences. It provides targeted guidance for overcoming the prevalent challenge of applying complex, evolving, or vaguely defined eligibility criteria during the evidence screening phase, a critical step for review integrity [1] [12].

Frequently Asked Questions (FAQs)

Category 1: Defining and Refining Eligibility Criteria

• Q1: Our review topic is highly interdisciplinary (e.g., combining toxicology, ecology, and chemistry). How can we create clear, applicable eligibility criteria?

  • A1: Develop criteria iteratively through pilot screening. As demonstrated in a case study on fecal coliform and land use, experts should independently screen a random sample of 50-100 studies, then meet to resolve discrepancies and refine the criteria wording [1]. This process should be repeated over 3-4 rounds until consensus and stability are achieved. Translate the final, explicit criteria into a structured prompt for any AI tool, clearly listing inclusion/exclusion conditions [1] [27].

• Q2: Our protocol's eligibility criteria seem too vague when applied to real studies. How should we proceed?

  • A2: Vagueness often arises from undefined key terms. Operationalize your criteria by creating a "codebook." For each criterion, provide:
    • A clear definition.
    • Positive and negative examples from your pilot screen.
    • Rules for borderline cases. This codebook becomes essential for training both human reviewers and AI models, ensuring consistent application [1].

Category 2: Implementing Screening with AI Assistance

• Q3: We want to use an AI model to assist with title/abstract screening. How do we set it up correctly?

  • A3: Follow a validated fine-tuning workflow [1]:
    • Prepare Training Data: Use your team's finalized pilot screening results (e.g., 130 studies labeled "Include"/"Exclude") [1].
    • Split Data: Divide this into training, validation, and test sets (e.g., 70, 20, and 40 studies) [1].
    • Fine-Tune Model: Use a base model like GPT-3.5 Turbo. Key hyperparameters include a low temperature (e.g., 0.4) for less random outputs, and top_p (e.g., 0.8) for focused token selection [1].
    • Account for Stochasticity: Run the model multiple times (e.g., 15 runs) per study and use the majority vote as the final decision to improve reliability [1].

• Q4: How do we write an effective prompt for the AI that encapsulates our complex criteria?

  • A4: Structure your prompt with clear instructions and repetition of key terms [27]. For example:
    • Role: "You are a systematic review screener in ecotoxicology."
    • Task: "Based on the following eligibility criteria, classify the study."
    • Criteria: List inclusions (PICO elements) and exclusions numerically.
    • Output Format: "Respond only with: 'INCLUDE' if [conditions met] OR 'EXCLUDE' if [condition not met]."
    • Text: "Here is the title and abstract: [PASTE TEXT]"

Category 3: Validating Performance and Ensuring Rigor

• Q5: How do we measure the performance and reliability of our AI-assisted screening process?

  • A5: Use standard agreement statistics on a held-out test set screened by experts [1].
    • Cohen's Kappa (κ): Measures agreement between the AI and a single expert reviewer.
    • Fleiss' Kappa: Measures agreement between the AI and multiple expert reviewers. A model showing "substantial agreement" (κ > 0.60) at the title/abstract stage can be considered a valid assistive tool [1]. Document the model's accuracy (e.g., 83% of relevant literature correctly identified) [27] and any systematic errors.

• Q6: Can we use AI for the full-text screening stage?

  • A6: Yes, but with caution. The process is similar to title/abstract screening but requires a refined prompt that instructs the model to consider information in the methods, results, and discussion sections [1]. Agreement with human reviewers may be lower ("moderate agreement") at this more complex stage, so AI outputs should be verified, not fully trusted [1]. Human review remains essential for final inclusion decisions.

Category 4: Managing Workflow and Disagreements

• Q7: How can digital tools help manage the screening workflow and team disagreements?

  • A7: Dedicated systematic review platforms (e.g., Covidence, Rayyan, EPPI-Reviewer) are designed for this [12]. They:
    • Facilitate independent dual screening by multiple reviewers.
    • Automatically highlight conflicts between reviewers' decisions for resolution.
    • Provide an audit trail of screening decisions.
    • Some integrate machine learning to prioritize studies likely to be relevant after initial screening [12].

• Q8: Reviewers from different disciplines disagree on applying criteria. How should we resolve this?

  • A8: This is a common interdisciplinary challenge [1]. Establish a pre-defined arbitration process:
    • The two initial reviewers discuss the conflict with reference to the codebook.
    • If unresolved, a third senior reviewer (or the entire team) makes a final decision.
    • Document the reason for the final decision and use it to further refine the codebook. AI-assisted screening can provide a consistent baseline application of criteria, helping to structure and mitigate these disagreements [1].

The table below summarizes key quantitative findings from recent case studies on AI-assisted screening in environmental research, providing benchmarks for expected performance.

Table 1: Performance Metrics from AI-Assisted Screening Case Studies

Study Focus	AI Model Used	Screening Stage	Key Performance Metric	Result	Source
Fecal coliform & land use	Fine-tuned GPT-3.5 Turbo	Title/Abstract	Agreement with human reviewers (Fleiss' Kappa)	Substantial agreement	[1]
Fecal coliform & land use	Fine-tuned GPT-3.5 Turbo	Full-Text	Agreement with human reviewers (Fleiss' Kappa)	Moderate agreement	[1]
Ecosystem condition indicators	GPT-3.5	Title/Abstract	Percentage of relevant literature correctly identified	83%	[27]

Detailed Experimental Protocol: AI-Assisted Screening Workflow Based on the methodology from [1], below is a step-by-step protocol for implementing an AI-assisted screening process.

Objective: To semi-automate the screening of literature for a systematic review, ensuring consistent application of eligibility criteria and improving efficiency.

Materials:

• A validated set of eligibility criteria and a codebook.
• A library of retrieved article citations and abstracts (e.g., from Scopus, Web of Science).
• Systematic review software (e.g., Covidence, Rayyan) or a reference manager.
• Access to an API for a large language model (e.g., OpenAI's GPT-3.5 Turbo or GPT-4).
• Statistical software (e.g., R, Python) for analysis.

Procedure:

• Pilot Screening & Criteria Finalization:
  • Randomly select 100-150 studies from your search results.
  • Have at least two expert reviewers independently screen these studies (title/abstract) against the draft criteria.
  • Convene to resolve conflicts and refine the wording of vague or inconsistently applied criteria.
  • Repeat for 3-4 rounds until consensus and stability are achieved. This final set is your "gold standard" criteria.

• Training Data Preparation:

  • Use the screening decisions from the finalized pilot round as labeled training data.
  • Split the data into three sets: Training (70%), Validation (15%), and Test (15%). Ensure balanced class labels (Include/Exclude) in the training set.

• AI Model Fine-Tuning & Prompt Engineering:

  • Format your eligibility criteria into a structured, unambiguous prompt.
  • Using the training set, fine-tune a base LLM (e.g., GPT-3.5 Turbo). Recommended hyperparameters include: epochs=3, batch_size=8, learning_rate=1e-5, temperature=0.4, top_p=0.8 [1].
  • Use the validation set to check for overfitting.

• Model Validation & Testing:

  • Apply the fine-tuned model to the held-out Test Set.
  • Calculate inter-rater agreement (Cohen's Kappa) between the AI and each human reviewer, and use Fleiss' Kappa for multi-reviewer agreement [1].
  • If agreement is substantial (κ > 0.60), proceed to screen the full corpus.

• Full Corpus Screening & Human Verification:

  • Use the model to screen all remaining uncategorized studies.
  • To mitigate the model's stochastic nature, run each study classification multiple times (e.g., 15 runs) and take the majority vote as the decision [1].
  • Crucially, all studies marked "Include" by the AI must be verified by a human reviewer at the next stage (full-text retrieval). AI is an assistive tool, not a replacement for expert judgment.

Visual Workflow: AI-Assisted Screening Process

The following diagram illustrates the integrated human-AI workflow for systematic review screening, from criteria development to final inclusion.

Diagram 1: AI-Assisted Systematic Review Screening Workflow (72 characters)

The Scientist's Toolkit: Research Reagent Solutions

This table catalogs essential digital tools and resources for managing the systematic review screening process, particularly when dealing with complex eligibility criteria.

Table 2: Key Digital Tools for Systematic Review Screening Automation

Tool Name	Category/Type	Primary Function in Screening	Key Consideration for Ecotoxicology
Covidence	Comprehensive SR Platform [12]	Manages the entire screening workflow: import, de-duplication, dual-blind screening, conflict resolution, and full-text review.	User-friendly interface facilitates teamwork among interdisciplinary reviewers. Subscription-based [12].
Rayyan	Comprehensive SR Platform	AI-assisted tool for collaborative title/abstract screening. Uses machine learning to predict relevancy and highlight conflicts.	Free tier available; useful for piloting AI-assisted screening on a budget.
EPPI-Reviewer	Comprehensive SR Platform [12]	Supports complex review types, data extraction, and synthesis. Offers text mining and machine learning classifiers.	High flexibility makes it suitable for complex, non-interventional reviews common in environmental sciences [12].
DistillerSR	Comprehensive SR Platform [12]	Web-based platform focusing on auditability and compliance for high-stakes reviews (e.g., regulatory).	Strong workflow management ensures reproducible application of complex criteria. Enterprise-focused pricing [12].
Python (scikit-learn, spaCy)	Programming Language / Libraries	Enables custom development of machine learning classifiers for screening based on your own training data.	Requires programming expertise. Offers maximum control for tailoring models to specific, niche terminology in ecotoxicology.
OpenAI API (GPT models)	Large Language Model API [1] [27]	Can be fine-tuned with domain-specific screening decisions to perform binary inclusion/exclusion classification.	As demonstrated in research, effective when properly fine-tuned and validated [1] [27]. Cost and data privacy must be managed.
PRISMA Statement	Reporting Guideline	A minimum set of items for reporting systematic reviews, including flow diagrams for screening.	Using PRISMA ensures transparency, which is critical when employing novel AI-assisted methods [12].

Technical Support Center: Troubleshooting AI/ML Tools for Systematic Review Screening in Ecotoxicology

This support center addresses common issues researchers face when implementing AI/ML tools to automate the screening of primary studies in ecotoxicology systematic reviews. The guidance is framed within a thesis on developing robust, domain-specific automation tools.

Frequently Asked Questions (FAQs)

Q1: Our model achieves high accuracy (~95%) on the training set but performs poorly (~60% recall) on new, unseen batches of ecotoxicology abstracts. What is the primary cause? A1: This is typically a training data quality and representativeness issue. The initial training corpus likely does not capture the full diversity of terminology, chemical names, and experimental designs present in the broader ecotoxicology literature.

Q2: How often should we re-calibrate or re-train our active learning screening model? A2: Implement iterative calibration at defined milestones. A standard protocol is to recalibrate after every 200-300 newly screened titles/abstracts, or whenever the topic composition of the incoming literature stream shifts significantly (e.g., moving from pesticide studies to pharmaceutical contaminants).

Q3: What is the optimal point for human-in-the-loop (HITL) review in the screening workflow to maximize efficiency? A3: HITL review is most effective as a continuous, integrated checkpoint. The key is to have the human expert review model predictions with low confidence scores and a random sample of high-confidence predictions to correct drift. This should be done in each calibration cycle.

Q4: The model is consistently misclassifying studies on "ecosystem services" as irrelevant. How can we fix this systematic error? A4: This is a domain-specific concept leakage problem. You must augment your training data with counterexamples. Manually identify and label 50-100 relevant studies that discuss ecosystem services in the context of toxicant impacts, and add them to your next calibration training batch.

Q5: What are the minimum annotation requirements to start an effective active learning process for study screening? A5: While starting can be iterative, research indicates a strong baseline requires a minimum of 200-300 dual-reviewed (included/excluded) references to seed the model. Prioritize annotating studies that are "edge cases" or semantically challenging.

Troubleshooting Guides

Issue: Rapid Performance Degradation (Concept Drift)

• Symptoms: Model precision/recall drops over time as screening of a large review progresses.
• Diagnosis: The literature corpus's thematic focus or terminology is shifting beyond the model's initial training.
• Solution:
  • Pause full automation.
  • Export the last 500 model-predicted records.
  • Have a human reviewer label a stratified random sample of 100 (50 high-confidence includes, 50 high-confidence excludes).
  • Calculate new performance metrics on this sample.
  • Use these newly labeled records, enriched with misclassified examples, to fine-tune the model.
  • Resume screening with the updated model.

Issue: High Volume of Low-Confidence Predictions

• Symptoms: The model classifies a very large percentage (e.g., >40%) of records with confidence scores between 0.4 and 0.6, requiring excessive manual review.
• Diagnosis: The model is uncertain due to ambiguous features. Common in ecotoxicology where broad terms like "effect," "impact," or "concentration" appear in both relevant and irrelevant studies.
• Solution:
  • Feature Engineering: Add domain-specific stop words (e.g., "socioeconomic," "economic cost") and expand the feature set to include n-grams (e.g., "adverse effect," "growth inhibition," "LC50").
  • Active Learning: Prioritize the manual screening of these low-confidence records. Use these new labels as the most valuable data for the next model retraining cycle, as they represent the decision boundary.

Issue: Bias Towards Frequently Studied Chemicals

• Symptoms: The model performs well for well-known contaminants (e.g., atrazine, bisphenol-A) but misses studies on emerging contaminants or less common chemical names.
• Diagnosis: Training data is skewed towards historically researched substances.
• Solution:
  • Create a chemical ontology lookup table. Integrate a list of known synonyms, CAS numbers, and common brand names for both historical and emerging contaminants.
  • Use this table to pre-process text, standardizing chemical names.
  • Proactively search for and label studies on a curated list of emerging contaminants to balance the training set.

Experimental Protocols & Data

Protocol 1: Benchmarking Training Data Quality

• Objective: Quantify the representativeness of an initial seed dataset for ecotoxicology screening.
• Methodology:
  • From your seed dataset (N=500 references), extract the top 100 unigrams and bigrams.
  • Perform a systematic search in a target database (e.g., PubMed, Web of Science) for your review topic. Randomly sample 500 references from the result set.
  • Extract top features from this broader corpus.
  • Calculate Jaccard similarity or cosine similarity between the feature vectors of the seed set and the broader corpus.
  • A similarity score <0.35 indicates poor representativeness and signals the need for broader seed data collection before model training.

Protocol 2: Iterative Calibration for an Active Learning Screener

• Objective: Systematically improve model performance during a live screening process.
• Methodology:
  • Start: Train initial model on seed data (e.g., 300 dual-reviewed references).
  • Screening Batch: Apply model to screen next 1000 unreviewed references.
  • HITL Sampling: Human reviews (a) all model-included references, (b) a 10% random sample of model-excluded references, and (c) 100 lowest-confidence predictions.
  • Performance Audit: Calculate work saved over sampling (WSS) at 95% recall and precision on the audited sample.
  • Retraining: Combine new labels from Step 3 with the existing training set. Retrain the model.
  • Loop: Return to Step 2. Repeat until screening is complete.

Table 1: Quantitative Impact of Iterative Calibration Cycles on Screening Performance

Calibration Cycle	Training Set Size	WSS@95% Recall	Precision	Relevant Studies Missed (Per 1000)
Initial (Seed)	300	42%	0.78	~50
After 1st Batch (200 new labels)	500	67%	0.85	~33
After 2nd Batch (200 new labels)	700	75%	0.88	~25
After 3rd Batch (200 new labels)	900	81%	0.91	~19

Table 2: Common Feature Engineering Solutions for Ecotoxicology Text

Problem Feature	Solution	Implementation Example
Ambiguous common words	Domain-specific stop word list	Add: "cost," "policy," "management," "society"
Critical dosage info	Regex pattern for key metrics	`(LC50	EC50	IC50	NOAEC	LOAEC)\s[=:]?\s\d+.?\d*`
Chemical name variants	Synonym normalization dictionary	"glyphosate" -> also map "N-(phosphonomethyl)glycine", "Roundup"
Organism terms	Taxon-specific word grouping	{"daphnia", "d. magna", "cladocera"} -> group tag: #FRESHWATER_INVERTEBRATE

Visualizations

Active Learning Screening Workflow with HITL

Iterative Model Calibration Loop

The Scientist's Toolkit: Essential Research Reagent Solutions

Table 3: Key Materials for Building an Ecotoxicology AI Screening System

Item	Function & Rationale
Dual-Annotated Seed Library	A foundational set of references (min. 300) where inclusion/exclusion decisions are made by two domain experts. Resolves ambiguity and provides gold-standard labels for initial model training.
Ecotoxicology Ontology / Thesaurus	A structured vocabulary (e.g., derived from ECOTOX, EPA terms) to map synonyms (e.g., "fish mortality" -> "lethality in piscines"), normalizing diverse terminology for the model.
Chemical Registry Lookup Table	A database linking chemical names, CAS numbers, and common trade names. Critical for identifying studies on the same contaminant referred to by different names.
Confidence Threshold Slider	A tool (software parameter) to adjust the prediction confidence score required for automatic exclusion. Allows tuning the balance between workload (WSS) and risk of missing relevant studies.
Stratified Random Sampling Tool	A script to select audit samples that ensure representation of high-confidence includes/excludes and low-confidence predictions. Enables efficient performance auditing.
Performance Metric Dashboard	Real-time visualization of Work Saved over Sampling (WSS) at various recall levels, precision, and relevance yield. Essential for monitoring drift and triggering recalibration.

Frequently Asked Questions (FAQs)

Q1: What is the most significant barrier to adopting automation tools for systematic review screening, and how can we overcome it? The most frequently cited barrier is a lack of knowledge, identified by 51% of surveyed practitioners [9]. This includes unfamiliarity with available tools and how to use them. Overcoming this requires structured training, as 72% of users are self-taught [9]. Research teams should seek institutional training, utilize online tutorials from software providers, and consult with information specialists to build competency.

Q2: At which stage of a systematic review are automation tools most commonly used? Automation tools are used most frequently during the screening stage. In a survey of systematic reviewers, 79% reported using tools specifically for screening titles and abstracts [9]. This is followed by their use in data extraction and critical appraisal.

Q3: Can automation tools reduce the time required for a systematic review? Yes. A significant majority (80%) of tool users report that these tools save them time [9]. Furthermore, over half (54%) believe that using automation tools increases the accuracy of their review process [9]. Properly implemented tools help manage the large volume of records typically encountered in interdisciplinary reviews.

Q4: How do I choose the right tool for an ecotoxicology systematic review that includes diverse study designs (e.g., in vivo, in vitro, in silico)? Select a tool that offers high customizability for inclusion/exclusion criteria and data extraction forms. Tools like Covidence, Rayyan, and DistillerSR are designed to handle varied study types [28] [29]. Prioritize tools that allow for complex, hierarchical screening questions to accurately appraise different experimental methodologies commonly found in ecotoxicology.

Q5: What are the most common reasons researchers abandon a specific automation tool? Tools are often abandoned due to cost, lack of desired features, or steep learning curves [9]. Rayyan (19%), Covidence (15%), DistillerSR (14%), and RevMan (13%) were cited as the most commonly abandoned tools [9]. Before committing, teams should utilize free trials to assess a tool's fit for their specific project needs and team workflow.

Troubleshooting Guides

Problem: Inconsistent Screening Decisions Among Reviewers

Symptoms: Low inter-rater reliability (Kappa score), frequent conflicts requiring third-party arbitration, final included study list that seems illogical or inconsistent with the protocol.

Solution:

• Pilot the Screening Form: Before beginning formal screening, all reviewers must independently screen the same small batch of 50-100 references using the draft criteria [30].
• Calculate Agreement and Calibrate: Measure inter-rater reliability. Discuss all conflicts in this pilot batch to clarify ambiguity in the written criteria. Revise the protocol and screening form language based on this discussion.
• Implement a "Calibration Threshold": Continue pilot screening in batches until the team achieves a pre-defined Kappa score (e.g., >0.8), indicating consistent understanding.
• Use Software Features: Leverage your tool's "conflict resolution" workflow to systematically highlight and resolve disagreements. Maintain a shared log of difficult decisions as precedent.

Problem: Managing an Overwhelming Volume of Search Results

Symptoms: Screening timeline becomes unmanageable, reviewer fatigue leads to errors, the team questions the scope of the research question.

Solution:

• Refine the Question: Return to the protocol. An overwhelming number of results often signals an unfocused review question [31]. Use frameworks like PICO to narrow the population, exposure, or outcome.
• Leverage Automation Features: Use your software's priority screening or machine learning ranking features if available (e.g., in SWIFT-ActiveScreener or EPPI-Reviewer) [12]. These can learn from your initial decisions and surface likely relevant studies.
• Apply Systematic "Not" Exclusions: After piloting, clearly define and apply specific, justifiable exclusions (e.g., publication date before a key regulatory change, non-peer-reviewed commentary, irrelevant population). Document these in the PRISMA flow diagram [32].

Symptoms: Missing key reports from regulatory agencies or dissertations; difficulty screening non-journal formats; duplication between database and grey literature searches.

Solution:

• Plan and Document: Define your grey literature sources (e.g., EPA reports, EFSA opinions, dissertation databases) in the protocol [31]. Use a structured search log for these sources just as you would for bibliographic databases.
• Deduplicate Systematically: After importing results from all sources (journal databases and grey literature), use your software's deduplication function. Manually check a sample to ensure accuracy, as title formatting can differ widely.
• Create Adapted Screening Forms: For grey literature documents, you may need a slightly different screening form that first assesses the document type (e.g., "Is this a primary research report or a summary review?") before applying standard eligibility criteria.

Problem: Handling Diverse Data Formats for Extraction

Symptoms: Data extraction forms cannot adequately capture findings from radically different study designs (e.g., a 96-hour LC50 from a fish assay vs. a gene expression profile from a microarray study).

Solution:

• Design Modular Extraction Forms: Instead of one monolithic form, create design-specific modules within your tool. All studies pass through a common first module (citation, study aim), then branch to an in vivo ecotoxicity, in vitro mechanistic, or computational module.
• Pilot Extensively: Pilot the extraction form on several studies of each design type. This reveals where fields are ambiguous or insufficient.
• Standardize Units at Entry: Define and enforce standard units (e.g., all concentrations in μM, all durations in hours) in the form design to avoid post-hoc harmonization errors.

Experimental Protocols for Key Tasks

Protocol 1: Establishing and Validating a Machine Learning (ML)-Assisted Screening Workflow

Objective: To integrate an ML-based prioritization tool into the title/abstract screening phase to improve efficiency while maintaining rigor.

Materials: A systematic review software with active learning capabilities (e.g., EPPI-Reviewer, SWIFT-ActiveScreener); a validated search results file (.RIS format); a team of at least two reviewers.

Procedure:

• Initial Seed Training: Import the deduplicated search results. Both reviewers will independently screen a common, randomly selected "seed" batch of 200-300 references.
• Train the Algorithm: After resolving conflicts in the seed batch, upload the definitive "include/exclude" decisions to train the tool's ML classifier.
• Prioritized Screening: The software will now rank the remaining references from most to least likely to be relevant. Reviewers screen in this order.
• Stopping Point Pilot: Periodically (e.g., after every 500 screened records), assess the yield. A pre-defined stopping rule (e.g., "stop when 100 consecutive records are excluded") can be tested.
• Validation Check: After screening all records deemed relevant by priority order, a random sample of the lowest-ranked records (e.g., 10% of the unscreened pile) must be screened manually to validate that no relevant studies were missed.

Diagram: ML-Assisted Screening Workflow

Protocol 2: Conducting a Calibration Exercise for an Interdisciplinary Review Team

Objective: To achieve a high level of consensus among reviewers with different disciplinary backgrounds (e.g., a toxicologist, an ecologist, and a computational biologist) before beginning formal screening.

Materials: Pre-written inclusion/exclusion criteria; a pilot library of 100 references deliberately selected to include clear includes, clear excludes, and ambiguous "edge cases"; screening software (e.g., Rayyan, Covidence); a shared document for notes.

Procedure:

• Blinded Pilot Screening: Each reviewer independently screens the entire 100-reference pilot library using the draft criteria.
• Quantitative Analysis: The lead reviewer calculates percent agreement and Cohen's Kappa for each reviewer pair.
• Structured Conflict Discussion: The team meets to review every conflict. The discussion focuses on the wording of the criteria that led to the different decision, not on defending a position.
• Protocol Refinement: Based on the discussion, revise the eligibility criteria to be more explicit. For example, change "studies on freshwater species" to "studies on fish or aquatic invertebrates residing in freshwater habitats (salinity < 0.5 ppt). Laboratory studies using reconstituted freshwater are included."
• Re-test (if necessary): If consensus remains low (<80% agreement), repeat the process with a new set of 50 references and the refined criteria until adequate agreement is achieved.

Comparative Analysis of Screening Tools

The table below summarizes key automation tools, their applicability to managing diverse ecotoxicology data, and user experience metrics based on survey data [9] [28] [29].

Table 1: Comparison of Systematic Review Automation Tools

Tool Name	Primary Use Case & Strengths	Cost Model	Reported User Adoption & Experience	Key Consideration for Interdisciplinary Data
Covidence	All-in-one platform for screening, extraction, and risk of bias. Strong collaborative features.	Annual subscription (free for Cochrane authors).	Most frequently cited "top 3" tool (45%). Commonly abandoned (15%) [9].	Highly structured workflow ensures consistency. Custom data extraction forms can be designed for varied study types.
Rayyan	Free, collaborative title/abstract screening. Intuitive interface with keyword highlighting.	Freemium model (free core features).	A top 3 tool for 22% of users. Most commonly abandoned tool (19%) [9].	Excellent for initial screening. May require exporting to other tools for complex data extraction from diverse designs.
DistillerSR	Enterprise-grade tool with powerful AI/ML features, audit trails, and robust compliance.	Monthly or annual subscription.	Cited as a top tool and abandoned by 14% of users [9].	High customizability is ideal for complex, protocol-driven reviews with multiple study designs. Learning curve can be steep.
JBI SUMARI	Supports 10 different review types (effectiveness, qualitative, scoping, etc.) beyond just interventions.	Annual subscription.	Part of the "Big Four" comprehensive tools [12].	Uniquely suited for reviews that mix quantitative and qualitative data from field studies, lab experiments, and models.
EPPI-Reviewer	Advanced tool with integrated machine learning ("priority screening") and support for complex synthesis.	Monthly per-review/user or institutional.	One of the "Big Four" comprehensive tools [12]. Open-source code.	ML prioritization is highly effective for large, interdisciplinary result sets. Powerful for mapping diverse evidence.

Table 2: Quantitative Insights on Tool Adoption and Impact [9]

Metric	Survey Result (%)	Implication for Practice
Users experiencing time savings	80%	Automation tools are a worthwhile investment for efficiency.
Users perceiving increased accuracy	54%	Tools support more reliable and consistent screening.
Lack of knowledge as a barrier	51%	Training is critical. Do not assume intuitive use.
Self-taught tool users	72%	Institutional or structured training can fill a major gap.
Tools most used during screening	79%	Screening is the primary pain point these tools address.

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Materials for Systematic Review Screening

Item	Function in the Screening Process
Reference Management Software (e.g., EndNote, Zotero, Mendeley)	Used to export search results from multiple databases into a single library, perform initial deduplication, and generate citation files (.RIS) for upload into screening software [32] [12].
Screening Software (e.g., Covidence, Rayyan)	Provides the digital collaborative workspace for independent title/abstract and full-text screening, conflict resolution, and progression tracking [28] [29].
PRISMA Flow Diagram Tool	A mandatory reporting item. The PRISMA diagram visually documents the flow of records through the screening phases, tracking numbers of included and excluded studies at each stage [32] [30].
Pre-defined Screening Form / Criteria	The operational protocol for reviewers. A clear, unambiguous document that translates the research question into specific, actionable questions about population, exposure, comparator, and outcome for each study type [30].
Inter-Rater Reliability (IRR) Calculator	A statistical tool (e.g., for Cohen's Kappa) used during the calibration phase to quantitatively measure agreement between reviewers before full screening begins, ensuring consistency [31].
Project Management Platform (e.g., Teams, Slack, Trello)	Facilitates asynchronous communication for review teams to discuss edge cases, update on progress, and share documents, which is crucial for managing long-term projects [31].

Team Management and Communication Workflow

Effective management of an interdisciplinary team is critical for consistent screening. The following diagram outlines a communication and decision-making structure to prevent common project derailments like scope creep, reviewer drift, and protocol violations [31].

Diagram: Interdisciplinary Review Team Communication Structure

This technical support center is designed to assist researchers in navigating the challenges of automating systematic review screening within ecotoxicology. A core thesis in this field posits that leveraging specialized software tools and formalized protocols is critical for handling the volume and complexity of environmental toxicity data while upholding the highest standards of reproducibility and transparent reporting [33] [34] [1]. The following guides address common technical and methodological issues.

Troubleshooting Guides & FAQs

Q1: My automated script for retrieving data from the EPA ECOTOX database failed. How do I diagnose the issue?

• Explanation & Cause: Automated data retrieval can fail due to changes in the source database's Application Programming Interface (API), network instability, incorrect query parameters, or package dependencies in your scripting environment (e.g., R) [33] [35].
• Solution: First, check for error messages in your console or log file. Verify your internet connection and ensure the database is accessible. Use packages like ECOTOXr in R, which are designed for reproducible access, and check their documentation for updates [33]. Re-run a simple, known-working query to isolate the problem. Maintain a log of your search queries, dates, and the number of records retrieved as an audit trail [36].

Q2: Our team is getting inconsistent results during the AI-assisted title/abstract screening phase. How can we ensure consistency?

• Explanation & Cause: Inconsistency often stems from ambiguous or poorly defined eligibility criteria, which are then interpreted differently by human reviewers or applied unstably by an AI model [1]. In interdisciplinary ecotoxicology reviews, terminology differences can exacerbate this [1].
• Solution: Before screening, conduct multiple calibration rounds with all reviewers. Screen a random sample of records independently, discuss disagreements, and refine the eligibility criteria iteratively until you achieve high inter-rater agreement (e.g., Cohen's Kappa > 0.6) [1]. Document the final, explicit criteria and use them to fine-tune any AI model. For AI tools, perform multiple runs and use majority voting to stabilize stochastic outputs [1].

Q3: How do I create a proper audit trail for my systematic review screening process?

• Explanation & Cause: An audit trail is a mandatory, documented record of every decision made during the review process. Without it, the work is not reproducible or transparent [33] [36].
• Solution: Systematically document the following in a spreadsheet or dedicated software (e.g., Covidence, Rayyan) [37]:
  • Search: Each database/platform searched, date of search, full search string, and number of hits [36].
  • Screening: The number of records at each stage (after de-duplication, title/abstract, full-text), with reasons for exclusion at the full-text stage clearly recorded [38] [36].
  • Tools: Note software names, versions (e.g., ECOTOXr, R, Python, AI model versions), and key parameters used [33] [1].

Q4: My systematic review protocol is complete. What are the most common pitfalls in reporting according to PRISMA 2020 guidelines?

• Explanation & Cause: PRISMA 2020 ensures complete and transparent reporting [39]. Common pitfalls include an incomplete flow diagram, insufficient detail on the search strategy, and not explaining exclusions at the full-text stage [34] [40].
• Solution: Use the official PRISMA 2020 Checklist and Flow Diagram template [39]. For the flow diagram, use software to generate accurate numbers from your audit trail. In the methods, state you followed PRISMA 2020, list all information sources (databases, registries, websites) with search dates, and provide a full search strategy for at least one database [37] [40].

Q5: The lab automation system for high-throughput ecotoxicity screening has stopped working. What's a systematic way to troubleshoot?

• Explanation & Cause: Lab automation failures can be mechanical (misaligned parts, clogged lines), electrical (no power), software-based, or due to human error (incorrect sample prep) [35] [41].
• Solution: Follow a structured diagnostic approach [35]:
  • Identify & Define: Pinpoint the exact failed step in the workflow.
  • Gather Data: Check system error logs, review recent changes to protocols or samples, and see if the issue is reproducible.
  • List Causes: Start with simple causes (power cord, reagent levels, sample air bubbles) before complex software issues [41].
  • Diagnose & Isolate: Run system diagnostics or a simplified test protocol.
  • Seek Help: Consult colleagues, online forums, and finally, the vendor's technical support [35].

Data Synthesis: Performance Metrics & Global Trends

Table 1: Performance Metrics of AI-Assisted Screening in a Systematic Review [1]

Screening Stage	Human-Human Agreement (Fleiss' Kappa)	AI-Human Agreement (Cohen's Kappa)	Key Note
Title/Abstract Screening	0.61 (Substantial)	0.62 (Substantial)	AI model fine-tuned with domain-specific criteria.
Full-Text Screening	0.42 (Moderate)	0.41 (Moderate)	Highlights complexity of full-text assessment.

Table 2: Global Distribution of Studies on Emerging Contaminants (2020-2024) [38]

Region	Percentage of Studies	Most Reported Contaminant Classes
Asia	37.05%	Microplastics, Antibiotics
Europe	24.31%	Personal Care Products, Endocrine Disruptors
North America	14.01%	Per- and Polyfluoroalkyl Substances (PFAS)
Africa	8.92%	Varied
South America	7.32%	Varied

Detailed Experimental Protocols

Protocol 1: Implementing an AI-Assisted Screening Workflow [1] This protocol outlines the integration of a large language model (LLM) to semi-automate the screening process.

• Team Formation: Assemble a team with domain experts (for screening and criteria development) and technical specialists (for model management).
• Pilot Screening & Criteria Calibration:
  • Reviewers independently screen a random pilot sample (e.g., 100-150 records).
  • Discuss conflicts and iteratively refine eligibility criteria over multiple rounds until consensus is stable.
• AI Model Fine-Tuning:
  • Translate finalized eligibility criteria into a structured prompt.
  • Use the pilot screening results (labeled "Include"/"Exclude") as a training dataset.
  • Fine-tune a model (e.g., ChatGPT-3.5 Turbo) using this data, adjusting hyperparameters (learning rate, epochs).
• Validation & Deployment:
  • Test the fine-tuned model on a held-out validation set of human-screened records. Calculate agreement statistics (Cohen's Kappa).
  • Deploy the model to screen the remaining corpus. To account for AI randomness, run the screening multiple times (e.g., 15) and take the majority vote.
• Quality Control: Have human reviewers screen a random subset of the AI-screened records to validate performance.

Protocol 2: Conducting a PRISMA-Compliant Systematic Review in Ecotoxicology [34] [40]

• Protocol Registration: Register the review protocol on a platform like PROSPERO or OSF to pre-specify methods and reduce bias.
• Question Formulation: Define a focused question using a structured framework (e.g., PECO: Population, Exposure, Comparator, Outcome).
• Search Strategy:
  • Search multiple databases (e.g., PubMed, Scopus, Web of Science, ECOTOX) with tailored Boolean strings [37] [38].
  • Supplement with grey literature searches. Document all search details for the audit trail [36].
• Screening Process:
  • Use reference management software to deduplicate records.
  • Perform title/abstract and full-text screening in duplicate, with conflicts resolved by consensus or a third reviewer.
• Data Extraction & Synthesis:
  • Extract data using a standardized, pre-piloted form.
  • Synthesize findings qualitatively (e.g., narrative synthesis) or quantitatively (meta-analysis) as appropriate. Assess the risk of bias in included studies.
• Reporting: Prepare the final report adhering to the PRISMA 2020 checklist, including the completed flow diagram [39].

Visualization of Workflows

AI-Enhanced Systematic Review Workflow

Structured Troubleshooting Decision Tree

Table 3: Key Tools for Automated Screening & Reproducible Research

Tool / Resource Name	Category	Primary Function in Ecotoxicology Reviews
ECOTOXr [33]	Data Curation R Package	Programmatically and reproducibly retrieves and subsets data from the US EPA ECOTOX knowledgebase.
PRISMA 2020 Statement [39]	Reporting Guideline	Provides an evidence-based minimum set of items for reporting systematic reviews and meta-analyses.
Covidence, Rayyan [37]	Screening Software	Online platforms for managing title/abstract and full-text screening in duplicate, with conflict resolution.
Fine-tuned LLM (e.g., ChatGPT) [1]	AI Screening Assistant	Augments human screening by applying consistent eligibility criteria to large volumes of text.
Zotero / EndNote [37]	Reference Manager	Manages citations, removes duplicates, and stores PDFs throughout the review process.
R / Python with Meta-analysis libraries	Statistical Software	Conducts statistical synthesis (meta-analysis), generates forest plots, and assesses heterogeneity.
Audit Trail Spreadsheet / Log [36]	Documentation	Records all decisions, search results, and exclusion reasons to ensure full transparency and reproducibility.

This technical support center is designed for researchers, scientists, and drug development professionals engaged in ecotoxicology systematic reviews. As the volume of scientific literature grows, teams increasingly turn to digital tools and artificial intelligence (AI) to automate the screening process. However, this integration introduces specific technical challenges related to data handling, team collaboration, and software constraints. The following guides and FAQs address these issues within the context of a broader thesis on automating systematic review screening, providing actionable solutions to keep your research on track [12] [1].

Data Import and Processing Errors

Importing search results from databases (e.g., Web of Science, Scopus) into screening platforms is a foundational step where errors can occur, potentially compromising your dataset before screening begins.

Troubleshooting Guide: Common Data Import Errors

The table below summarizes frequent import errors, their causes, and resolution strategies, compiled from technical documentation and user communities [42] [43] [44].

Table: Common Data Import Errors and Resolutions for Systematic Review Screening

Error Category	Typical Error Message / Code	Likely Cause	Recommended Resolution
Schema Mismatch	"Missing or mismatched columns," "No mappings found for query" [43] [44]	CSV column headers don't match the tool's expected field names (e.g., "Author" vs. "First Author").	Use the tool's data mapping interface to manually align columns. If available, leverage AI-powered column matching features [44].
Data Type/Format Error	"Could not parse date," "Invalid number value," "Invalid boolean value" [42] [43]	Date formats differ (MM/DD/YYYY vs. DD-MM-YYYY), numbers contain text characters, or fields expect true/false values.	Standardize data formats in your source file before import. Use the import tool's preview to correct values individually [42] [44].
Lookup/Reference Failure	"Association record not found," "Lookup reference could not be resolved" [42] [45]	Attempting to import or link records (e.g., articles linked to journals) where the referenced entity doesn't yet exist in the system.	Import entities in the correct order (e.g., journal records before article records). Ensure unique identifiers (IDs) in your file match those in the system [45].
Duplicate Detection	"Duplicate: this record already exists" [43] [45]	The import file contains records identical to existing ones based on system rules (e.g., same title and author).	Review and temporarily disable strict duplicate detection rules for the import if appropriate, then re-enable them [45].
File Structure Issues	"Malformed CSV," "Unable to read from the data source" [43] [44]	Extra line breaks, inconsistent delimiters, special characters, or file corruption.	Re-save the file as a UTF-8 encoded CSV. Use a robust import tool that handles various file types and encodings gracefully [44].

FAQs: Data Import

Q1: After importing my search results, hundreds of records are missing. What happened? A: This is often due to deduplication settings or filtering rules applied during the import. First, check the import log or summary report for details on excluded records [42] [43]. Common reasons are:

• Automatic deduplication: The platform may have removed records with identical titles or DOIs.
• Field mapping failures: Records where critical fields (like "Title") couldn't be mapped were likely skipped.
• File parsing limits: Some tools stop parsing a file after encountering too many errors in initial rows [43].

Solution: Review the error file, correct your source data (e.g., ensure proper formatting), and re-import. Before final import, perform a test with a small record batch [42].

Q2: How can I prevent "lookup reference" errors when importing articles and their source journals? A: This error occurs when your data has relational dependencies. The system cannot create an article linked to "Journal X" if "Journal X" isn't already in its database [45].

• Staged Import: First, import a list of unique journal names to create the journal entities. Then, import the articles, using the now-established journal IDs or names for linking.
• Use Unique Identifiers: Where possible, use persistent identifiers like the ISSN for journals to ensure consistent matching.

Q3: My import fails with a generic "job failed" error. How do I diagnose it? A: Generic errors require checking system logs. Look for a correlation ID in the error message, which support teams use to trace the failure [43]. Common underlying issues include:

• Empty Data Source: The file path is wrong, or the uploaded file is empty [43].
• Permission Problems: The system service account lacks write permissions for the target database.
• Size Limitations: The file may exceed your platform's row or size limit for a single import [43].

Experimental Protocol: Standardized Data Import and Cleaning

A reproducible import process is critical for review integrity. The following protocol is adapted from best practices in data management [42] [1] [44].

Objective: To clean and import bibliographic search results into a systematic review screening tool (e.g., Covidence, EPPI-Reviewer) without data loss or corruption.

Materials: Bibliographic export file(s) (e.g., .ris, .csv, .enw), a reference manager (e.g., Zotero, Mendeley), a text editor or spreadsheet application, and access to your chosen screening platform.

Methodology:

• Export: From each database (Scopus, PubMed, etc.), export the complete search results. Choose a format compatible with both your reference manager and screening tool (.ris is widely supported).
• Consolidate & Deduplicate:
  • Import all files into your reference manager.
  • Use the manager's deduplication function to remove exact duplicates. Manually review suspected duplicates based on title and author.
  • Export the deduplicated library as a single file.
• Pre-Import Cleaning (if using CSV):
  • Open the exported file in a spreadsheet application.
  • Standardize Formats: Ensure date columns follow a single format (YYYY-MM-DD is recommended). Check author name columns for consistent separation (e.g., "Lastname, Firstname").
  • Handle Special Characters: Remove or encode non-UTF-8 characters that may cause parsing errors.
  • Check for Empty Required Fields: Identify records missing critical data (Title, Abstract) for later review.
• Test Import:
  • Create a new project in your screening platform.
  • Import the cleaned file. If the platform allows a dry run or validation, use it.
  • Systematically address all errors reported in the log, correcting them in your master file.
• Final Import: Once the test import succeeds with zero critical errors, perform the final import. Save the final, cleaned import file alongside the original exports as part of your review audit trail.

Collaboration and Conflict Management

Systematic reviews require multiple reviewers to screen independently, leading to inevitable disagreements. Managing these conflicts constructively is key to maintaining progress and team morale [46].

Conflict Resolution Framework

Effective conflict resolution transforms friction into collaboration. The following strategies are recommended for research teams [46] [47].

Table: Conflict Resolution Strategies for Review Teams

Conflict Scenario	Root Cause	Immediate Action	Long-Term Solution
Disagreement on Inclusion/Exclusion	Differing interpretation of eligibility criteria.	Blind Re-review: Both reviewers re-assess the article, noting the specific criterion in dispute.	Refine Criteria: Clarify the wording in the protocol. Use AI-assisted screening on a sample to highlight ambiguous patterns [1].
Workload Imbalance	One reviewer progresses slower, causing bottlenecks.	Redistribute Tasks: Temporarily reassign batches of records to maintain flow.	Set Clear Milestones: Use project management features in tools like Covidence to set and track weekly screening targets.
Protocol Adherence vs. Pragmatism	Debate over strictly following the protocol versus making a pragmatic exception.	Third-Party Arbitration: Involve the principal investigator or a third reviewer to make a binding decision based on the protocol's intent.	Document Deviations: Any agreed-upon exception must be formally documented as a protocol amendment to ensure reproducibility.

FAQs: Collaboration Conflicts

Q1: My co-reviewer and I consistently disagree on screening articles about a specific ecotoxicological method. How can we resolve this? A: Persistent disagreement on a specific topic often indicates ambiguous eligibility criteria. Follow this process:

• Pause & Discuss: Halt screening on new articles. Together, review a sample of 10-20 conflicted articles.
• Identify the Pattern: Determine the exact point of disagreement (e.g., "Does study X qualify as a 'field study' per our criterion 3?").
• Refine the Protocol: Collaboratively rewrite the problematic criterion to be more explicit. Incorporate examples of "include" and "exclude" from your sample.
• Re-calibrate: Use the updated criteria to independently re-screen the sample. Measure inter-rater agreement (e.g., Cohen's Kappa). Repeat until acceptable agreement is achieved (>0.6) [1].

Q2: Our team is distributed across time zones. What tools and practices can prevent collaboration delays? A: Leverage asynchronous collaboration features and clear communication rules [48].

• Tool Use: Choose platforms with built-in commenting @mentioning (like Rayyan or Covidence) to flag disputes without needing a meeting.
• Establish Protocols: Set a rule that flagged conflicts must be addressed within 48 hours. Use a shared spreadsheet or project board to track the status of disputed records.
• Record Decisions: Every resolved conflict must result in a brief note in the tool explaining the final decision, creating a searchable audit trail for the entire team.

Q3: How can AI assist in resolving screening conflicts? A: AI-assisted screening tools can act as a consistent, third "reviewer" to help resolve disputes [1].

• Predictive Prioritization: Tools like SWIFT-ActiveScreener or EPPI-Reviewer's ML prioritize records most likely to be relevant, allowing teams to focus discussion on the most ambiguous articles [12].
• Consistency Check: In the methodology tested by [1], a fine-tuned ChatGPT model showed substantial agreement with human reviewers. You can run a batch of conflicted articles through a calibrated AI model. Its consistent (though not infallible) output can provide a baseline for discussion and help identify which of your criteria are most subject to interpretation.

Diagram: Conflict Resolution Workflow for Dual Screening. This chart outlines the recommended path for resolving disagreements between reviewers, incorporating optional AI assistance and a final arbitration step to ensure consistent decisions [46] [1].

Software Limitations and AI Integration

While digital tools significantly accelerate reviews, understanding their limitations—particularly regarding AI functionality—is crucial for their responsible use [12] [1].

Understanding AI-Assisted Screening

AI in systematic review tools typically uses machine learning (ML) or large language models (LLMs) to predict an article's relevance. A 2025 study in environmental evidence provides a clear experimental protocol for integrating an LLM [1].

Experimental Protocol: Fine-Tuning an LLM for Title/Abstract Screening

Objective: To assess the feasibility of a fine-tuned ChatGPT-3.5 Turbo model for performing title and abstract screening in a systematic review on ecotoxicology.

Materials:

• Software: Access to OpenAI API (for GPT-3.5 Turbo), programming environment (e.g., Python, R).
• Data: A labeled dataset of at least 100-150 articles from your review, screened by human experts, with binary labels (Include/Exclude).
• Reference: The methodology follows the workflow described by [1].

Methodology:

• Prepare Training Data: From your labeled articles, create a structured dataset. Each entry should contain the article's title and abstract and the correct eligibility decision. Split the data into training (e.g., 70%), validation (e.g., 15%), and test (e.g., 15%) sets.
• Fine-Tune the Model: Using the OpenAI API, fine-tune the gpt-3.5-turbo model on your training set. Key hyperparameters from the cited study [1] include:
  • Epochs: 3-4 (to prevent overfitting).
  • Learning Rate: Use the recommended default from OpenAI.
  • Batch Size: Configure based on your dataset size and API constraints.
• Prompt Engineering: Integrate your eligibility criteria into a system prompt. For example: "You are a systematic review assistant. Based on the following eligibility criteria: [List criteria]. Determine if the provided title and abstract should be included. Answer only 'Include' or 'Exclude'."
• Evaluate Performance: Run the fine-tuned model on the held-out test set. Calculate performance metrics against the human consensus:
  • Cohen's Kappa (κ): Measures agreement between the AI and a human reviewer.
  • Sensitivity/Recall: Proportion of true included articles correctly identified.
  • Specificity: Proportion of true excluded articles correctly identified.
• Implement in Workflow: The study [1] used a majority-voting approach (15 model runs per article) to stabilize the stochastic output. The final decision was based on the majority result. Use the model as a second screener or a tie-breaker, not a fully autonomous agent.

FAQs: Software and AI Limitations

Q1: Can I fully automate the screening process with AI? A: No. Full automation is neither reliable nor currently considered methodologically sound for a definitive systematic review [12] [1]. AI should be used as an assistive technology:

• Prioritization: It can rank articles from most to least likely to be relevant, allowing reviewers to find included studies faster (work sampling).
• Second Pass: It can act as a consistent second screener, but its results must be verified by a human, especially for articles it predicts as "exclude" [1].
• Conflict Insight: As noted, its pattern can help identify criteria ambiguity.

Q2: My screening tool's AI keeps suggesting I exclude articles that I think are relevant. What should I do? A: This indicates a potential mismatch between the AI's model and your specific review question. Most integrated AI tools are trained on general biomedical literature and may perform poorly on niche ecotoxicology topics [12].

• Retrain/Calibrate: If the tool allows it, feed it a sample of your already-screened articles to improve its predictions for your project.
• Override and Document: The human reviewer's decision is final. Consistently overriding the AI in one direction is a critical piece of methodological information that should be reported in your paper's methods section.

Q3: We are using a "one-stop-shop" tool like Covidence. What are its main limitations for complex ecotoxicology reviews? A: Comprehensive tools like Covidence, DistillerSR, and EPPI-Reviewer are validated for intervention reviews but may have limitations for environmental sciences [12]:

• Complex Data Extraction: They may lack custom forms for extracting diverse data types common in ecotoxicology (e.g., toxicokinetic parameters, sediment characteristics).
• Risk of Bias Tools: Built-in checklists (e.g., ROB-2) are for clinical trials. Assessing risk of bias in observational environmental studies may require custom forms.
• AI Features: Their integrated ML models are often trained on biomedical literature, which may limit out-of-the-box accuracy for ecotoxicology.

Table: Key Research Reagent Solutions for Automated Screening

Tool / Resource Name	Category	Primary Function in Screening	Key Consideration
Covidence, DistillerSR, EPPI-Reviewer [12]	Comprehensive Screening Platform	End-to-end management of screening, full-text review, data extraction, and quality assessment in a collaborative online workspace.	Subscription costs; AI features may be add-ons. Best for standard review types but adaptable.
Rayyan	Screening Platform	Free-to-use tool for efficient title/abstract screening with AI-powered prioritization and conflict highlighting.	A good entry-level option, but may lack advanced data extraction and project management features.
SWIFT-ActiveScreener [12]	AI-Powered Prioritization	Uses active machine learning to continuously learn from reviewer decisions and rank unscreened records by predicted relevance.	Can be integrated into other workflows; significantly reduces screening workload.
Python/R with OpenAI/LLM Libraries [1]	Custom AI Integration	Allows for custom fine-tuning and deployment of LLMs (like GPT) for tailored screening assistance, as per the experimental protocol.	Requires programming expertise; offers maximum flexibility for methodological research.
PRISMA 2020 Statement	Reporting Guideline	The essential checklist and flow diagram framework for transparently reporting your systematic review.	Using a tool that auto-generates a PRISMA flowchart from your screening data is a major efficiency gain.
Zotero, Mendeley [12] [1]	Reference Management	Centralized management of search results, deduplication, and export to screening platforms.	Critical for the pre-screening data cleaning and organization phase.

Evaluating Performance: Benchmarks, Comparisons, and Emerging Technologies

Technical Support Center: Troubleshooting Guides & FAQs

This support center addresses common challenges in implementing and evaluating automated screening tools for systematic reviews in ecotoxicology. The guidance is framed within a thesis on advancing automation to manage the rapidly expanding volume of toxicological literature, thereby accelerating chemical safety and risk assessments [18] [49].

Section 1: Interpreting Performance Metrics

Q1: We ran an automated screening tool that reported 95% recall and 70% work saved. Is this result reliable enough to stop manual screening early? A: A 95% recall is strong, indicating the tool identified most relevant studies. However, before stopping, you must verify the absolute number of missed studies (false negatives). In a large corpus, even 5% can be significant. We recommend a validation step: manually screen all records excluded by the algorithm for a random subset (e.g., 10%) of your total references. If no relevant studies are found in this excluded set, you can proceed with higher confidence. Note that tools like ASReview have shown mean workload savings of 83% when aiming for 95% recall (WSS@95) [50].

Q2: Our tool achieved high precision (>90%) but low recall (<60%). What does this mean for our review, and how can we fix it? A: This pattern means your tool is correctly including relevant studies (low false positives) but is missing too many relevant ones (high false negatives). This is a critical issue for a systematic review, as missing studies compromise validity. The problem often lies in the training set or feature definitions.

• Solution: Expand and diversify your initial training set. If using a PECO-based model, check if your rules are too strict [18]. For machine learning tools, include more examples of "relevant" studies that represent different sub-topics or ways of describing key concepts. Retrain the model and test recall on a held-out validation set.

Q3: How do we measure and improve screening consistency between human reviewers and the AI tool? A:* Screening consistency is measured by inter-rater reliability metrics like Cohen's Kappa. A recent study using LLM-generated PICOS summaries achieved a Kappa of 99.8% between human reviewers, indicating near-perfect agreement [51].

• To Improve Consistency:
  • Standardize Inputs: Use AI to generate structured summaries (e.g., PICOS) for each study. This gives both human reviewers the same, clear information layout [51].
  • Calibration Exercises: Before the main screening, have all reviewers (human and AI) screen the same small batch of studies. Discuss discrepancies to align understanding of inclusion criteria.
  • Define Clear Rules: Ambiguity in criteria is the main source of inconsistency. Use frameworks like PECO (Population, Exposure, Comparator, Outcome) to operationalize your review question [18] [37].

Table 1: Key Performance Metrics for Screening Automation

Metric	Formula	What It Measures	Target in Ecotoxicology
Work Saved (WS)	1 - (TP + FP) / N [18]	Reduction in records requiring manual review.	High variability (30%-96%) [18] [50]. Prioritize high recall first.
Recall (Sensitivity)	TP / (TP + FN) [52]	Ability to identify all relevant studies.	Near 100% is critical. Must minimize false negatives.
Precision	TP / (TP + FP) [52]	Proportion of selected records that are relevant.	Often trades off with recall. >80% is efficient [52].
Specificity	TN / (TN + FP)	Ability to correctly exclude irrelevant studies.	Reported alongside precision; 99.9% achieved with AI assistance [51].
F1 Score	2 * (Precision * Recall) / (Precision + Recall) [52]	Harmonic mean of precision and recall.	Useful balanced score when comparing models.
Cohen's Kappa	-	Agreement between raters (human-human or human-AI).	>0.8 indicates strong agreement [51].

Section 2: Experimental Setup & Protocol Design

Q4: What is the minimum viable protocol for testing a new screening tool on our ecotoxicology review data? A: Follow this standardized validation protocol:

• Data Preparation: Start with a fully screened, labeled dataset (e.g., a completed review). Divide it into a training pool (e.g., 75%) and a held-out test set (25%) [52].
• Simulated Screening: Use the tool's "simulation" mode. Seed the model with an initial batch of relevant and irrelevant studies from the training pool (e.g., 10-20 records).
• Active Learning Loop: Simulate the tool ranking the remaining training pool. Iteratively "feed" it the next highest-ranked record and update the model, mimicking real-world use.
• Performance Calculation: After processing the training pool, apply the final model to the held-out test set. Calculate recall, precision, and work saved at different screening milestones (e.g., after screening 50% of records) [50].
• Reporting: Document the Work Saved over Sampling (WSS) at recall levels of 95% and 100% [50].

Q5: How do we set up a PECO-based automated screening experiment as described in the literature? A:* The protocol from [18] involves rule-based extraction:

• Define PECO Elements: Precisely define the Population, Exposure, Comparator (or Confounder), and Outcome for your review. Use controlled vocabularies (e.g., ECOTOX terms) [49].
• Implement Extraction: Use a text mining tool (like GATE) with dictionaries and semantic rules to identify mentions of these elements in titles and abstracts [18].
• Apply Screening Rules: Test different inclusion rules (e.g., requiring Exposure+Outcome (EO) vs. all four PECO terms). The study found the EO rule was most effective, excluding 93.7% of studies with 98% recall [18].
• Validate: Compare the algorithm's inclusions/exclusions against the gold-standard manual screening results to calculate performance metrics.

Table 2: Comparison of Common Screening Automation Tools & Approaches

Tool/Approach	Core Methodology	Typical Work Saved	Best For	Considerations
PECO Rule-Based [18]	Extraction of predefined elements (Population, Exposure, etc.)	Up to 93.7% [18]	Reviews with very well-defined, consistently reported key elements.	Requires upfront rule development; depends on abstract reporting quality.
Research Screener [50]	Machine learning (simulation suggests active learning)	60% - 96% [50]	Researchers seeking a semi-automated tool with strong published validation.	Performance validated across multiple real and simulated reviews.
Rayyan [50]	NLP n-grams & support vector machines	Avg. ~49% (WSS@95) [50]	Collaborative, manual screening with ML assistance for prioritization.	Free, web-based, and good for team collaboration.
ASReview [50]	Active learning with multiple model choices	67% - 92% (WSS@95) [50]	Researchers who want an open-source, state-of-the-art active learning platform.	Highly customizable; supports simulation for benchmarking.
LLM (PICOS) [51]	Large Language Model generates structured summaries	~75% workload reduction [51]	Accelerating manual screening by providing consistent, extracted data points.	Assists human reviewers; does not fully automate decision-making.

Section 3: Technical Implementation & Data Issues

Q6: Our text mining tool fails to extract key terms from older scanned PDFs. How do we handle poor-quality text data? A: This is a common data pipeline issue.

• Preprocessing: Invest in Optical Character Recognition (OCR) software to convert scanned PDFs to machine-readable text. Manually verify a sample for accuracy.
• Text Enhancement: For abstracts with poor formatting, use sentence segmentation tools to restore structure.
• Vocabulary Mapping: Older studies may use outdated terminology. Expand your extraction dictionaries to include historical synonyms (e.g., "biocide" vs. "pesticide").

Q7: The machine learning model performs well on one review topic but poorly when applied to another. Why? A:* This is due to lack of domain adaptation. Models trained on one corpus learn specific linguistic patterns that may not transfer.

• Solution: Do not use a model trained on a different domain without retraining. Implement a rapid retraining protocol:
  • Start with the pre-trained model.
  • Screen a small, representative batch of your new corpus (e.g., 50-100 studies).
  • Use these new labels to fine-tune or retrain the model via active learning. This allows the tool to adapt to the new topic's vocabulary [50].

Section 4: Tool-Specific Troubleshooting

Q8: When using an active learning tool (e.g., ASReview, Rayyan), how do we decide when to stop screening? A:* This is a strategic decision balancing risk and effort.

• Use the WSS Metric: Monitor the tool's dashboard for the Work Saved over Sampling at 100% recall (WSS@100). This estimates the screening fraction needed to find all relevant studies [50].
• Apply a Stopping Rule: A common, conservative rule is to stop manual screening after you have consecutively screened a large number of irrelevant records ranked highly by the model. For example, after screening 100-200 consecutive records without a new inclusion, you can pause.
• Final Validation: As in Q1, always manually screen a random sample of the excluded records to estimate the potential for missed studies.

Q9: How effective are new Large Language Models (LLMs) like ChatGPT for full automation, and what are the risks? A:* Current research advises against full automation with LLMs due to risks of missing studies ("hallucination" of exclusion reasons). Their most effective use is human-in-the-loop assistance. A 2025 study found that providing reviewers with an LLM-generated structured PICOS summary led to a 75% reduction in screening time while achieving perfect (100%) sensitivity [51].

• Recommended Protocol: Use the LLM to process titles/abstracts and output a consistent summary of Population, Intervention/Exposure, Comparator, Outcome, and Study Design. The human reviewer then makes the inclusion decision based on this clear, structured data, drastically improving speed and consistency [51] [53].

Experimental Protocols for Key Cited Studies

Protocol 1: Validating a PECO-Based Screening Rule [18]

• Objective: To evaluate if automated extraction of study characteristics (PECO) can effectively screen studies.
• Materials: A gold-standard, completed systematic review dataset with inclusion/exclusion labels.
• Steps:
  • For each study abstract, run a text-mining algorithm (e.g., using GATE) to detect mentions of Population (P), Exposure (E), Comparator/Confounder (C), and Outcome (O).
  • Apply a screening rule (e.g., "include if both E and O are present").
  • Compare algorithm decisions to gold-standard labels. Calculate True Positives (TP), False Negatives (FN), etc.
  • Calculate Recall = TP/(TP+FN) and Work Saved = 1 - (TP+FP)/Total Studies.
• Expected Outcome: The study found the "E and O" rule saved 93.7% of screening work with 98% recall [18].

Protocol 2: Benchmarking an Active Learning Tool [50]

• Objective: To measure the workload savings of a tool like Research Screener or ASReview.
• Materials: A labeled dataset split into training and test sets.
• Steps:
  • In the tool's simulation mode, seed the model with a small random sample (e.g., 10 studies) from the training set.
  • Initiate the active learning loop, where the tool prioritizes the next most relevant record from the training pool. This record is then "labeled" and added to the training data.
  • Continue until all records in the training pool are "screened."
  • Plot a recall curve (percentage of total relevant found vs. percentage of total records screened).
  • Calculate WSS@95 and WSS@100 from the curve.
• Expected Outcome: Tools like ASReview showed mean WSS@95 of 83% and WSS@100 of 61% across multiple reviews [50].

Visualizing Workflows and Logic

Diagram 1: Decision logic for PECO-based automated screening rules.

Diagram 2: Systematic review workflow integrating screening automation.

The Scientist's Toolkit: Essential Research Reagents & Materials

Table 3: Key Tools and Resources for Automated Screening in Ecotoxicology

Item	Category	Function & Relevance	Example / Source
Labeled Review Datasets	Data	Gold-standard data for training & benchmarking algorithms.	Your own completed reviews; public repositories like CADIMA.
Text Processing Engine	Software	Extracts and processes text from abstracts/PDFs for analysis.	General Architecture for Text Engineering (GATE) [18], spaCy.
Screening Automation Software	Tool	The core platform that implements ML or rules for screening.	Research Screener [50], ASReview, Rayyan, DistillerSR.
Large Language Model (LLM) API	Tool	Generates structured summaries (PICOS) to assist human screeners.	OpenAI GPT, Google Gemini, open-source models (Mistral) [51] [53].
Ecotox-Specific Databases	Database	Provides controlled vocabularies and data for defining PECO terms.	EPA ECOTOX Knowledgebase [49], Comptox Chemicals Dashboard [49].
Reference Manager	Software	Manages search results, removes duplicates, and facilitates screening.	EndNote [18], Zotero, Mendeley [37].
Validation Framework	Protocol	Standard method to evaluate tool performance before full deployment.	Work Saved over Sampling (WSS) metric & simulation mode [50].

Technical Support Center

This support center provides troubleshooting guidance for common technical issues encountered while using systematic review automation platforms within ecotoxicology research projects. The guidance is framed within experimental protocols for tool evaluation.

Frequently Asked Questions (FAQs) & Troubleshooting

Q1: During the initial import of search results from databases (e.g., PubMed, Scopus) into Covidence, many records are failing to import. What could be the cause and solution?

• Issue: This is often due to formatting inconsistencies or duplicate PMID/DOI fields in the RIS or CSV file exported from the database.
• Troubleshooting Steps:
  • Verify Export Format: Ensure you have exported the records from your database in a supported format (RIS is most reliable).
  • Pre-process the File: Open the RIS file in a text editor. Check for and remove any header/footer text that is not part of a standard RIS entry.
  • Use a Deduplication First Strategy: Import the file into a reference manager like EndNote or Zotero first, deduplicate there, and then export a clean RIS file for import into Covidence.
  • Split Large Files: If you have >10,000 records, split the file into batches of 5,000 and import sequentially.

Q2: In DistillerSR, the AI Rank tool does not appear to be prioritizing relevant ecotoxicology studies. How can I improve its performance?

• Issue: The AI model requires a sufficient number of "seed" decisions (typically 50-100 relevant and irrelevant records) to learn the specific criteria of your review.
• Troubleshooting Protocol:
  • Initial Manual Screening: Conduct a pilot screen of at least 200-300 records manually, marking them as "Included" or "Excluded" according to your PICO/PECO criteria.
  • Train the AI: Navigate to the AI Rank settings and initiate or retrain the model using your completed pilot screenings.
  • Validate Performance: Screen the next 100 records manually but note the AI's prediction. Calculate the percentage where AI rank aligned with your decision. Performance typically improves after screening ~500 records.
  • Refine Keywords: Ensure your exclusion criteria keywords (e.g., "human," "clinical trial") are correctly listed in the project's "Word Frequency" exclude list to aid the AI.

Q3: When collaborating on Rayyan, some team members see different conflict resolution flags or their progress is not syncing. What should we check?

• Issue: This is almost always related to refresh or cache issues within the web application, or differing user permissions.
• Troubleshooting Steps:
  • Hard Refresh: Instruct all users to perform a hard refresh (Ctrl+F5 or Cmd+Shift+R) of their browser window.
  • Clear Cache: If the issue persists, have users clear their browser cache for the Rayyan website.
  • Check Permissions: The project owner should verify that all collaborators have the correct level of access (e.g., "Member" vs. "Viewer"). Only "Members" and "Admins" can resolve conflicts.
  • Synchronization: Ensure all users have clicked the "Sync" button (circular arrow icon) to force an update with the central server.

Q4: The EPPI-Reviewer machine learning classifier is producing a high number of false positives during title/abstract screening for a topic on "PFAS aquatic toxicity." How can I recalibrate it?

• Issue: The classifier's probability threshold may be set too low, or the training set may not be representative.
• Experimental Protocol for Recalibration:
  • Access the Model: Go to the 'Machine Learning' section for your screening task.
  • Review Training Set: Manually review the records used to train the model. Ensure they are correctly labeled and represent the diversity of your topic (e.g., include various PFAS compounds, freshwater/marine species).
  • Adjust the Threshold: Locate the classification threshold slider. Increase the threshold (e.g., from 0.5 to 0.7) to make the classifier more conservative, reducing false positives.
  • Test Iteratively: Apply the new threshold to a batch of unscreened records. Manually check a sample of records marked as "included" by the AI to assess the new false positive rate. Repeat step 3 as needed.

Q5: Across all platforms, the automated deduplication process is missing a significant number of duplicates. What is the standard manual check protocol?

• Issue: No automated deduplication algorithm is 100% accurate, especially with inconsistent author lists or journal abbreviations.
• Standardized Experimental Protocol for Manual Deduplication:
  • Export Suspect Lists: Export lists of records sorted by "Author + Year" and "Title" (first 5 words) from your platform.
  • Cross-Reference in Spreadsheet: Import these lists into a spreadsheet (e.g., Excel). Use conditional formatting to highlight rows with identical author/year and similar titles.
  • Manual Inspection: For highlighted records, compare the full metadata (journal, volume, pages, DOI). Key identifiers are DOI and PMID; a match confirms a duplicate.
  • Platform-Specific Action: Return to your platform and manually flag or remove the confirmed duplicates. In Covidence, use the "Merge duplicates" function.

Platform Comparison & System Requirements

Table 1: Core Functionality & Technical Specifications Comparison

Feature	Covidence	DistillerSR	Rayyan	EPPI-Reviewer
AI Automation	Limited to priority screening	AI Rank, text mining, auto-labeling	ML-assisted screening	Advanced machine learning classifiers, topic modeling
De-duplication	Automatic + manual merge	Automatic + manual review	Automatic + manual review	Automatic + sophisticated manual tools
Collaboration	Real-time, role-based	Real-time, audit trail, QA tools	Real-time, conflict highlighting	Real-time, supports large teams
Export Formats	RIS, CSV	CSV, XML, PDF	RIS, CSV, Excel	CSV, specialized report formats
Primary Access	Web-based	Web-based	Web-based & Mobile App	Web-based
Cost Model	Subscription (per reviewer/year)	Subscription (per project/user)	Freemium (paid for advanced features)	Institutional license / Subscription

Table 2: Experimental Setup & Resource Requirements

Item	Function in Systematic Review Screening
Reference File (RIS/ENW)	Standardized input containing bibliographic data of search results.
PICO/PECO Protocol	Defines inclusion/exclusion criteria; the essential "reagent" for training AI and guiding screeners.
Validation Set (Gold Standard)	A subset of records (~500) manually screened by all reviewers to measure inter-rater reliability and AI accuracy.
Deduplication Log	A spreadsheet tracking all merged or removed duplicate records for auditability.
Codebook / Tagging Dictionary	A pre-defined list of tags (e.g., "Endocrine disruptor," "Chronic exposure") for consistent data extraction.

Experimental Workflow Visualizations

Workflow for Systematic Review Screening with AI Platforms

AI Training and Prediction Process in Screening

Technical Support Center: AI-Assisted Screening for Environmental Systematic Reviews

This technical support center provides targeted guidance for researchers implementing AI-assisted screening tools within environmental systematic reviews (SRs), a core methodology for synthesizing evidence in fields like ecotoxicology. The content is framed within a thesis investigating tools for automating systematic review screening to enhance the rigor and efficiency of evidence synthesis in environmental health and toxicology.

Troubleshooting Guide: Common Technical and Methodological Issues

Problem 1: Low Inter-Rater Agreement Between AI and Human Screeners

• Symptoms: Poor Cohen's Kappa scores during validation; high rates of discordance on specific article types (e.g., modeling studies, interdisciplinary reports).
• Diagnosis & Solution: This often stems from poorly defined or inconsistently applied eligibility criteria. Refine your criteria through iterative calibration rounds with human reviewers before AI training [1]. Translate the final criteria into a clear, structured prompt for the AI, explicitly defining key concepts and exclusion rules [27]. For stochastic models, run multiple inference cycles (e.g., 15 runs) and use the majority vote to determine final inclusion/exclusion [1].

Problem 2: AI Model Overlooks Relevant Studies or Includes Too Many Irrelevant Ones

• Symptoms: Low recall (misses relevant papers) or low precision (includes too many irrelevant papers).
• Diagnosis & Solution: This is typically a training data issue. Ensure your fine-tuning dataset is balanced and representative of the literature landscape. It should include an equal number of relevant ("include") and irrelevant ("exclude") examples, validated by domain experts [1]. Review misclassified articles to identify patterns and refine your prompt or training data accordingly.

Problem 3: Inconsistent Results Across Different Screening Stages

• Symptoms: AI performs well on title/abstract screening but poorly on full-text screening, or vice versa.
• Diagnosis & Solution: Use stage-specific prompts and training. Eligibility criteria application differs between stages. Develop and fine-tune separate model prompts for title/abstract screening (focused on general relevance) and full-text screening (focused on methodological details and results) [1]. Use a dedicated, smaller set of expert-labeled full-text articles for full-text model validation.

Problem 4: Handling Interdisciplinary Terminology and Concepts

• Symptoms: AI fails to recognize synonymous terms across fields (e.g., "fecal coliform" vs. "E. coli" vs. "thermotolerant coliform" in hydrology vs. public health literature).
• Diagnosis & Solution: Actively build domain knowledge into the prompt. Expand your prompt to include a glossary of key terms and their variants across relevant disciplines [1]. Use the iterative prompt development process to "sharpen" key research terms, ensuring the AI understands the core multidimensional concept you are targeting [27].

Frequently Asked Questions (FAQs)

Q1: What are the validated performance metrics for AI screening tools in environmental reviews? A: Performance varies by model and domain. In a case study on ecosystem condition, GPT-3.5 correctly identified 83% of relevant literature [27]. Another case study on land use impacts reported substantial agreement (Kappa) at the title/abstract stage and moderate agreement at the full-text stage between AI and human experts [1]. Comparative studies show automation can reduce screening time for certain tasks from 42 hours (manual) to 12 hours (automated) while maintaining similar error rates [54].

Q2: How does the time investment for setting up AI-assisted screening compare to the time saved? A: The initial investment is significant. It requires time for team training, iterative criteria development, prompt engineering, and model fine-tuning [1]. However, this cost is front-loaded. For reviews involving screening hundreds or thousands of articles, the time savings in the screening phase itself are substantial and increase with the volume of literature [54]. The efficiency gain also allows for broader search strategies and more comprehensive reviews.

Q3: What is the biggest barrier to adopting these automation tools? A: A survey of systematic reviewers found that lack of knowledge about the tools' existence and capabilities was the most frequent barrier to adoption, cited by 51% of respondents [9]. Other barriers include distrust in the tool's accuracy and a preference for traditional manual methods [54].

Q4: Can AI completely replace human reviewers in the screening process? A: No. Current best practice uses AI as a screening assistant, not a replacement. The AI handles the initial bulk screening, but human experts are crucial for defining the protocol, training/validating the model, resolving ambiguous cases, and making final inclusion decisions. This hybrid approach maintains rigor while improving efficiency [1] [27].

Q5: What are the essential components of a team conducting an AI-assisted systematic review? A: A successful team requires integrated expertise [55]:

• Domain Experts (2-3): To define research questions, eligibility criteria, and validate outputs.
• Review Methodologist: To ensure adherence to PRISMA/Cochrane standards [1].
• AI/Technical Specialist: To manage model fine-tuning, API integration, and data processing.
• Information Specialist/Librarian: To design and execute comprehensive search strategies [55].

Experimental Protocols & Methodologies

Core Protocol: Fine-Tuning an LLM for Title/Abstract Screening [1]

• Team Assembly & Criteria Development: Form a team of 3+ domain experts. Conduct 3-4 rounds of independent, duplicate screening on a random sample of 100-150 articles. Discuss discrepancies after each round to refine and solidify eligibility criteria.
• Training Data Creation: From the consensus-labeled articles, create a balanced dataset (e.g., 35 "Include," 35 "Exclude"). Split this into training, validation, and test sets.
• Prompt Engineering: Translate the final eligibility criteria into a structured prompt. The prompt should clearly state the review's objective, inclusion/exclusion criteria, and the required output format (e.g., "Yes/No").
• Model Fine-Tuning: Use the training set to fine-tune a base LLM (e.g., ChatGPT-3.5 Turbo). Key hyperparameters to optimize include:
  • Epochs: 3-5 (to avoid overfitting).
  • Learning Rate: Use a low learning rate (e.g., 1e-5) for stable adjustments.
  • Batch Size: Adjust based on available computational memory.
• Stochastic Inference: Due to model randomness, perform multiple inference runs (e.g., 15) per article in the validation/test set. Use the majority vote as the final decision.
• Validation & Metrics: Evaluate performance on the held-out test set against expert labels. Calculate Cohen's Kappa (for agreement with a single expert) and Fleiss' Kappa (for agreement among multiple experts and the AI) [1].

Data Presentation: Performance and Efficiency Metrics

Table 1: Performance Metrics from AI-Assisted Screening Case Studies

Case Study Focus	AI Model Used	Key Performance Metric	Agreement Level with Humans	Source
Land Use & Fecal Coliform	Fine-tuned GPT-3.5 Turbo	Title/Abstract Screening	Substantial Agreement (Kappa)	[1]
Land Use & Fecal Coliform	Fine-tuned GPT-3.5 Turbo	Full-Text Screening	Moderate Agreement (Kappa)	[1]
Ecosystem Condition Indicators	GPT-3.5	Literature Screening	83% Correct Selection	[27]

Table 2: Time Efficiency Comparison: Manual vs. Automated Screening Tasks [54]

Systematic Review Task	Manual Team Time	Automation Team Time	Time Saved	Note on Error Rate
Run search, deduplicate, screen titles/abstracts & full text, assess risk of bias	2493 min (~42 hrs)	708 min (~12 hrs)	~71% reduction	Error rates were comparable or lower for automation in most tasks.

Visualizations: Workflows and Relationships

Diagram 1: AI-Assisted Screening Workflow

Diagram 2: Prompt Optimization Logic

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Tools & Resources for AI-Assisted Systematic Reviews

Tool/Resource Name	Category	Primary Function in AI-Assisted Review	Key Consideration
Large Language Model (LLM) API (e.g., OpenAI GPT, Anthropic Claude)	AI Engine	Core model for fine-tuning and performing classification/screening based on custom prompts.	Cost, data privacy policies, and fine-tuning capabilities are critical selection factors.
Systematic Review Automation Platforms (e.g., Rayyan, Covidence, DistillerSR)	Screening Management	Platforms to manage the screening process, often now integrating AI features to prioritize articles or suggest exclusions.	45% of surveyed reviewers use Covidence; 22% use Rayyan [9]. Assess AI feature maturity.
Bibliographic Reference Manager (e.g., Zotero, EndNote)	Reference Management	Essential for deduplication, storing full texts, and managing citations throughout the review process [1].	Must handle large libraries (1000+ references) and allow for export/import with screening platforms.
Statistical Software (e.g., R, Python with pandas)	Data Analysis	Calculate agreement statistics (Cohen's Kappa), analyze performance metrics, and manage training/validation datasets [1].	R/Python scripts are necessary for custom analysis beyond platform reporting.
PRISMA & Cochrane Guidelines	Methodological Framework	Provide the essential standards for conducting and reporting rigorous systematic reviews, which must be maintained when implementing AI [1].	The AI process must be transparently reported in the review's methods section.

For researchers in fields like ecotoxicology, conducting systematic reviews is essential but notoriously labor-intensive, particularly during the study screening phase [56]. While dedicated software tools exist to manage this process [57], the emergence of Large Language Models (LLMs) like ChatGPT presents a transformative opportunity: automating screening with a customized, domain-aware assistant. This technical support center focuses on the practical application of fine-tuning these general-purpose LLMs to create specialized tools for ecotoxicology evidence synthesis, addressing common challenges and providing clear protocols.

Technical Support & Troubleshooting Guide

This guide addresses specific, technical issues you may encounter when fine-tuning an LLM for systematic review screening.

Q1: My fine-tuned model is generating inconsistent screening decisions or "hallucinating" reasons for inclusion/exclusion. What steps should I take to improve reliability?

• Primary Cause & Solution: This is typically a data quality and alignment issue. Your training data must be unambiguous. Refine your dataset by ensuring each example consists of a clear article title/abstract and a correct, well-justified decision based on predefined PICO/PECO criteria [58]. Implement a two-stage tuning process: first, use Supervised Fine-Tuning (SFT) on high-quality, human-labeled examples to teach the task structure [58]. Second, employ Direct Preference Optimization (DPO). Create pairs of model outputs for the same input where one response is more accurate and factually grounded than the other. Training with these pairs directly steers the model toward preferred, reliable outputs [58].
• Advanced Troubleshooting:
  • Check for Contradictions: Audit your training set for conflicting labels on similar abstracts.
  • Temperature Setting: During inference, set the model's temperature parameter to a low value (e.g., 0.1) to reduce creativity and increase determinism [59].
  • Output Parsing: Force the model to structure its output in a strict format (e.g., "Decision: INCLUDE/EXCLUDE. Reason: [pre-defined criterion code]") to minimize free-text hallucinations.

Q2: I have limited computational resources (e.g., a single GPU with ≤24GB memory). Can I still fine-tune a large model like Llama 3 or GPT-2 for my project?

• Primary Cause & Solution: Yes, using Parameter-Efficient Fine-Tuning (PEFT) methods. Full fine-tuning is computationally prohibitive [60]. The recommended solution is QLoRA (Quantized Low-Rank Adaptation) [60].
  • Process: First, load your base model in a highly memory-efficient 4-bit quantized format. Then, train tiny, low-rank "adapter" matrices that are added to the model's layers. The original 99%+ of the model weights remain frozen, drastically cutting memory use [60].
  • Example Protocol: Use libraries like Hugging Face PEFT and bitsandbytes. A 7-billion parameter model can be fine-tuned on a single 24GB GPU using QLoRA, whereas full fine-tuning would require multiple high-end GPUs [60].

Q3: After fine-tuning on my ecotoxicology dataset, the model has become worse at general language understanding or forgets its original instruction-following capability. How can I prevent this "catastrophic forgetting"?

• Primary Cause & Solution: This occurs when the tuning process over-updates the model's core weights. The solution is to use Low-Rank Adaptation (LoRA) or QLoRA, which are designed to mitigate this risk [60] [58]. By freezing the base model and only training small adapter layers, the fundamental knowledge is preserved. After training, you can even merge the adapters back into the base model for a standalone, improved model, or keep them separate for modular use [60].

Q4: My domain (ecotoxicology) uses highly specialized terminology. How can I effectively teach the model this jargon and its context?

• Primary Cause & Solution: The general pre-training corpus lacks niche domain terms. Combine fine-tuning with Retrieval-Augmented Generation (RAG) [61].
  • Fine-Tune for Fundamentals: Use SFT to teach the model the screening task format and basic scientific reasoning.
  • Augment with RAG: Create a vector database of key ecotoxicology review articles, glossaries, and your review protocol. When the model screens a new abstract, the RAG system retrieves the most relevant context from this database and provides it to the model, grounding its decision in authoritative text [61]. This is more efficient than trying to encode all knowledge into the model's parameters.

Comparison of Systematic Review Automation Tools

The table below contrasts traditional systematic review software with the emerging paradigm of custom fine-tuned LLMs, based on analyzed features [56] [62] [57].

Feature	Traditional Review Software (e.g., Covidence, DistillerSR) [56] [57] [63]	Custom Fine-Tuned LLM (e.g., ChatGPT, Llama)
Core Function	Manages the workflow and collaboration of human screeners [57] [63].	Automates the cognitive screening decision for each article.
Learning Ability	Uses simple keyword highlighting or basic ML for prioritization; rules are static [63].	Adapts and improves from examples; understands context and synonyms.
Customization	Configurable forms, workflows, and labels [57].	Deeply customizable to specific domains, protocols, and team criteria via fine-tuning.
Handling Ambiguity	Low; relies on human judgment for complex cases.	Moderate to High; can infer relevance based on learned patterns, but requires human oversight.
Primary Cost	Financial (annual subscription fees) [62] [63].	Computational/Expertise (GPU resources, AI/ML engineering skill).
Best For	Standardized workflow management, team collaboration, and audit trails [57].	Accelerating screening throughput for large reviews, or handling domain-specific language.

This protocol outlines a methodology for creating a screening assistant, based on established fine-tuning pipelines [60] [58] [64].

1. Objective To fine-tune a pre-trained LLM to accurately classify scientific abstracts as "Include" or "Exclude" based on a defined set of ecotoxicology-focused PECO criteria.

2. Materials & Dataset Preparation

• Base Model: A publicly available instruct-tuned model (e.g., Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.2).
• Training Data: A minimum of 500-1000 high-quality, human-labeled examples are recommended to start [64]. Each example is a text prompt structured as: "### Instruction: Screen this abstract for a systematic review on [Your Topic]. The eligibility criteria are: [List PECO]. ### Abstract: [Title. Abstract text]. ### Response: Decision: [INCLUDE/EXCLUDE]. Reason: [Concise reason linked to criteria]."
• Data Splits: Divide your dataset into training (80%), validation (10%), and test (10%) sets.

3. Fine-Tuning Procedure using QLoRA This efficient method is ideal for limited resources [60].

• Model Quantization: Load the base model in 4-bit precision using the bitsandbytes library to reduce memory footprint [60].
• Apply LoRA Configuration: Configure LoRA to target the key projection matrices (q_proj, v_proj) in the model's attention layers. Use a low rank (r=8) and a scaling parameter (lora_alpha=32) [60].
• Training Arguments:
  • Learning Rate: Use a higher rate than full fine-tuning (e.g., 2e-4) as fewer parameters are updated [60].
  • Epochs: 3-5 epochs to avoid overfitting on smaller datasets.
  • Batch Size: Set as high as your GPU memory allows (e.g., 4-8).
• Training: Use the Hugging Face Trainer API with the SFT (Supervised Fine-Tuning) objective on your prepared prompt dataset [58].

4. Validation & Testing

• Monitor the validation loss during training to detect overfitting.
• On the held-out test set, calculate standard performance metrics: Precision, Recall, F1-score, and Cohen's Kappa (to measure agreement beyond chance with human screeners).
• Perform error analysis on misclassified abstracts to identify if failures are due to domain knowledge gaps, criterion ambiguity, or reasoning errors.

The Scientist's Toolkit: Essential Research Reagents

This table details the key "research reagents" – the software, data, and hardware components – required for a fine-tuning experiment.

Item	Function & Specification	Relevance to Experiment
Pre-trained Base Model	The foundational LLM (e.g., Llama 3, GPT-2). Provides general language and reasoning capabilities to build upon. Must be selected based on size, license, and instruction-following ability.	The core "material" being modified. An instruction-tuned model is preferable as a starting point [58].
Domain-Specific Dataset	Curated collection of (abstract, decision, reason) pairs. This is the critical reagent that teaches the model your task. Quality and consistency are paramount [64].	Directly determines the skill and reliability of the final fine-tuned model.
Fine-Tuning Library (PEFT)	Software library like Hugging Face's peft. Implements efficient methods like LoRA and QLoRA [60].	Enables the fine-tuning experiment to be feasible on limited academic compute resources.
GPU Hardware	Graphics Processing Unit with sufficient VRAM (e.g., NVIDIA A100, RTX 4090). Required for accelerated model training.	The "lab equipment" providing the computational power. QLoRA can reduce requirements to a single consumer-grade GPU [60].
Vector Database	Database for storing text embeddings (e.g., Chroma, Weaviate). Used for implementing RAG [61].	Optional but recommended for providing the model with up-to-date, external knowledge sources during screening.

Workflow and Architecture Diagrams

Diagram 1: Systematic Review Workflow with LLM Integration This diagram contrasts the traditional screening workflow with a pathway augmented by a fine-tuned LLM assistant.

Diagram 2: Custom LLM Fine-Tuning & Deployment Pipeline This diagram outlines the end-to-end technical process for creating and deploying a custom screening assistant.

Technical Support & Troubleshooting Center

This technical support center provides targeted troubleshooting guides and FAQs for researchers integrating AI tools into systematic review (SR) screening, with a focus on ecotoxicology and environmental evidence synthesis.

Frequently Asked Questions (FAQs)

Q1: What are the first steps to begin screening with an AI tool? Begin by clearly defining and documenting your review's eligibility criteria with your team. Convert these criteria into a structured prompt for the AI. Start by manually screening a small, random batch of articles (e.g., 50-100) with multiple reviewers to establish a "gold standard" dataset for training or validating the AI model [1]. This step is crucial for calibrating the tool to your specific research question.

Q2: My AI tool is excluding too many potentially relevant studies. How can I make it more inclusive? This indicates low recall (sensitivity). First, analyze the excluded studies to identify patterns. The issue likely stems from your eligibility criteria or prompts being too narrow [27]. Broaden keyword definitions, use more synonyms, and explicitly instruct the model to be "overly permissive" during the title/abstract screening phase, as is recommended in manual processes [65]. In tools like ASReview, you can adjust the classification threshold to prioritize recall over precision [22].

Q3: The AI is including too many irrelevant studies, creating more work. How can I improve precision? This is common, especially in early rounds. Precision often improves during the full-text screening phase with more refined criteria [65]. For the AI, refine your prompts by adding clear exclusion clauses and examples of irrelevant studies [27]. If using a trainable model, iteratively correct its errors on a validation set; this "relevance feedback" helps the model learn. Tools like RobotAnalyst are designed for this iterative learning process [22].

Q4: How do I evaluate if the AI is performing well enough to trust? Do not rely on the tool's output alone. Standard practice is to measure agreement between the AI and human reviewers. Use Cohen's Kappa (for 2 raters) or Fleiss' Kappa (for 3+) on a held-out test set of articles [1]. Performance benchmarks from meta-analyses can guide you: in medical SRs, AI models prioritizing maximum recall achieved a combined recall of 0.928, while those maximizing precision achieved a combined precision of 0.461 [66]. For environmental reviews, a case study using GPT-3.5 correctly selected 83% of relevant literature [27].

Q5: What are the most common technical errors in AI-assisted screening setups?

• Poor Training Data: Using a small, biased, or inconsistently labeled dataset for training.
• Prompt Drift: Changing eligibility criteria or prompt wording mid-screening without retraining or re-calibrating the model, leading to inconsistent decisions [1].
• Ignoring Stochasticity: LLMs can give different outputs for the same input. Best practice is to run queries multiple times (e.g., 15 runs) and take the majority vote as the final decision [1].
• Tool Misapplication: Using a model fine-tuned on clinical abstracts for ecotoxicology screening without domain adaptation.

Q6: Can I fully automate the screening process? No. Current consensus is that manual screening is still indispensable for final verification [66]. AI is best used as a "human-in-the-loop" system to prioritize workload and reduce manual screening burden, not to replace reviewers. Your role shifts from screening every item to validating the AI's work and resolving uncertain cases.

Troubleshooting Common Performance Issues

Table 1: Diagnosing and Resolving Common AI Screening Performance Issues

Symptom	Likely Cause	Diagnostic Check	Corrective Action
Low Recall(Missing relevant studies)	Overly strict prompts or criteria; model trained on unrepresentative data.	Manually review a sample of excluded records. Calculate recall against a human-screened test set.	Broaden prompts with synonyms and inclusive language [27]. Retrain with more inclusive examples. Adjust model threshold.
Low Precision(Too many irrelevant inclusions)	Vague exclusion criteria; prompts lack specificity; early-stage model.	Calculate precision. Check if included studies violate a specific, unstated exclusion rule.	Add explicit negative examples to prompts ("Exclude studies that only mention..."). Retrain model with corrected labels on irrelevant studies.
Inconsistent Decisions	Uncontrolled model randomness; drifting eligibility criteria.	Run the same article through the model multiple times. Review screening logs for criteria changes.	For LLMs, use a low temperature setting (e.g., 0.4) and employ majority voting from multiple runs [1]. Document and freeze criteria before bulk screening.
High Disagreement with Human Reviewers	Ambiguous eligibility criteria; interdisciplinary terminology gaps.	Calculate inter-rater agreement (Kappa) among humans first. Analyze discrepancies for systematic misunderstandings.	Refine criteria definitions with clear boundaries. Involve domain experts to align terminology. Use these resolved discussions to refine AI prompts [1].

Experimental Protocols for Validation

To ensure robustness when implementing an AI screening tool, follow this validation protocol adapted from recent research [1] [66]:

1. Protocol: Creating a Benchmark Dataset

• Objective: To create a reliable "gold standard" set for training and testing the AI model.
• Method:
  • From your total search results, randomly select a pilot sample (e.g., 100-200 articles).
  • At least two domain expert reviewers screen this pilot set independently, applying the eligibility criteria.
  • All conflicts are resolved through discussion or a third arbitrator.
  • The final, consensus-labeled pilot set is split into three: Training (e.g., 70%), Validation (e.g., 15%), and Test (e.g., 15%) sets.

2. Protocol: Fine-Tuning a Large Language Model (LLM)

• Objective: To adapt a general-purpose LLM (like GPT-3.5 Turbo) for specialized screening tasks.
• Method [1]:
  • Prompt Engineering: Translate eligibility criteria into a structured prompt, specifying the input (title/abstract) and required output (Include/Exclude with reason).
  • Model Training: Use the training set to fine-tune the base model. Key hyperparameters include:
    • Epochs: 3-5 (to avoid overfitting).
    • Learning Rate: A small value (e.g., 1e-5) for gradual weight updates.
    • Batch Size: Balishes training speed and memory (e.g., 8).
  • Validation: Use the validation set to tune hyperparameters and prompt wording.
  • Stochasticity Control: Set a low temperature (e.g., 0.4) for deterministic outputs. For critical decisions, run the final model on each article 15 times and take the majority vote.

3. Protocol: Performance Evaluation & Reporting

• Objective: To quantitatively assess the AI model's reliability.
• Method:
  • Run the finalized AI model on the held-out Test set.
  • Compare AI decisions to the human gold standard. Generate a confusion matrix.
  • Calculate key metrics: Recall (Sensitivity), Specificity, Precision (PPV).
  • Calculate Inter-rater Agreement between the AI and the human reviewer(s) using Cohen's Kappa.
  • Calculate Work Saved over Sampling (WSS) to estimate efficiency gains. WSS@95% recall is a common metric, indicating the percentage of screening saved while capturing 95% of relevant studies [66].

Visualization of Workflows & Architectures

AI-Assisted Systematic Review Screening Workflow

Human-in-the-Loop AI Screening System Architecture

Available Tools & Platforms

Table 2: Overview of AI-Assisted Systematic Review Screening Tools [22]

Tool Name	Access	Key AI/Methodology	Best For	Integration Note
ASReview	Free, Open-Source	Active Learning (Human-in-the-loop)	Teams starting with AI screening; high transparency needs.	Can be run locally; supports custom models.
Rayyan	Freemium, Web-Based	Machine Learning classifiers, keyword highlighting.	Collaborative teams needing a unified platform for all screening stages.	Cloud-based; easy to use but less customizable.
Abstrackr	Free, Web-Based	Machine Learning with relevance feedback.	Projects where reviewers can iteratively train the model during screening.	Semi-automated; requires user interaction.
RobotAnalyst	Free, Web-Based	Text mining & topic modelling for prioritization.	Exploring and categorizing large, unstructured literature sets.	Focuses on search and prioritization.
SWIFT-Review	Free, Desktop	Active Learning & Natural Language Processing (NLP).	Complex reviews requiring topic modeling and iterative query building.	Developed for chemical risk assessment.
PICO Portal	Freemium, Web-Based	NLP for deduplication and keyword highlighting.	Teams following PICO framework closely; intuitive interface.	Intelligent automation for workflow tasks.

The Scientist's Toolkit: Research Reagent Solutions

In the digital experiment of AI-assisted screening, the "reagents" are software, data, and computational resources.

Table 3: Essential Digital Research Reagents for AI-Assisted Screening

Item	Function/Description	Example/Note
Reference Management Software	Stores, deduplicates, and manages bibliographic records from database searches. Essential for feeding clean data into AI tools.	Zotero, EndNote [1].
Gold Standard Training Set	A manually screened, consensus-labeled set of articles. This is the critical reagent for training, validating, and benchmarking AI model performance.	Typically 100-300 articles, split into training/validation/test sets [1].
Fine-Tuned Language Model	A pre-trained LLM (the base reagent) adapted to your specific screening task via prompt engineering and fine-tuning on your gold standard data.	GPT-3.5 Turbo fine-tuned with environmental study abstracts [1].
Statistical Analysis Environment	Software for calculating performance metrics (recall, precision, Kappa) and statistical validation of the AI's output.	R Studio (with 'mada' package for meta-analysis) [1] [66], Python (scikit-learn).
Automation Pipeline Scripts	Code that connects different steps: data export from reference manager -> preprocessing -> AI model query -> results aggregation.	Custom Python/R scripts, or built-in workflows in tools like ASReview.
Collaborative Screening Platform	A cloud-based platform that manages the screening workflow, records decisions, resolves conflicts, and often integrates AI prioritization.	Rayyan, Covidence [65], PICO Portal [22]. These platforms are the "lab bench" where the digital experiment is run.

Conclusion

The automation of systematic review screening in ecotoxicology is no longer a luxury but a necessity to manage the scale and complexity of modern research. As demonstrated, a suite of sophisticated software tools and AI methodologies can dramatically reduce manual workload while enhancing methodological rigor and transparency. Success hinges on a strategic approach: selecting the right tool for the project's scope, carefully implementing and validating automated processes, and maintaining human oversight for complex decisions. For biomedical and clinical research, the advancements in ecotoxicology offer a parallel path forward. The integration of AI for screening, coupled with structured, interoperable databases, presents a model for accelerating evidence synthesis across disciplines. Future directions point towards greater AI autonomy, seamless integration of living review models, and the development of domain-specific large language models. Embracing these tools will be crucial for generating timely, high-quality evidence to inform environmental protection and public health decisions.

References

• 1. AI-assisted evidence screening method for systematic reviews ... [https://environmentalevidencejournal.biomedcentral.com/articles/10.1186/s13750-025-00358-5]
• 2. Review Accessible methods and tools to estimate chemical ... [https://www.sciencedirect.com/science/article/abs/pii/S0269749124008236?via%3Dihub]
• 3. Completing A Systematic Review in Hours instead of ... [https://arxiv.org/html/2504.14822v1]
• 4. Glossary of key terms associated with environmental ... [https://www.ciimar.up.pt/Arcopol_2016/page_22.htm]
• 5. 1.3 Key concepts and terminology in ecotoxicology [https://fiveable.me/ecotoxicology/unit-1/key-concepts-terminology-ecotoxicology/study-guide/e5Rxwu5cb2ZQWKbt]
• 6. Ecotoxicology (ECOTOX) Knowledgebase Resource Hub [https://www.epa.gov/comptox-tools/ecotoxicology-ecotox-knowledgebase-resource-hub]
• 7. Interdisciplinary Toxicology → Term [https://pollution.sustainability-directory.com/term/interdisciplinary-toxicology/]
• 8. A systematic review of Explainable Artificial Intelligence ... [https://www.sciencedirect.com/science/article/pii/S277266222300070X]
• 9. Systematic review automation tools improve efficiency but ... [https://www.sciencedirect.com/science/article/abs/pii/S0895435621002080]
• 10. Automated screening of research studies for systematic ... [https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-018-0724-7]
• 11. Guide to Troubleshooting Industrial Control and ... [https://forum.digikey.com/t/guide-to-troubleshooting-industrial-control-and-automation-equipment/42390]
• 12. Digital Tools to Support the Systematic Review Process [https://pmc.ncbi.nlm.nih.gov/articles/PMC12035789/]
• 13. SWIFT-Active Screener [https://www.sciome.com/swift-activescreener/]
• 14. ECOTOX Knowledgebase Quick Guide [https://epa.figshare.com/articles/online_resource/ECOTOX_Knowledgebase_Quick_Guide/26764618]
• 15. EPI Suite™-Estimation Program Interface [https://www.epa.gov/tsca-screening-tools/epi-suitetm-estimation-program-interface]
• 16. Ecotox REACH 2025 Conference: Key Takeaways [https://www.complianceandrisks.com/cr-news/ecotox-reach-2025-conference-key-takeaways-on-reach-2-0-clp-revision-and-pfas-restrictions/]
• 17. Ecotoxicology & Risk Assessment [https://complianceservices.com/services/ecotoxicology-and-risk-assessment/]
• 18. Automated screening of research studies for systematic ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC5918752/]
• 19. Formulating the Question, Eligibility Criteria, and Protocol [https://ktbooks.ca/knowledge-translation-in-health-care/summarized-book-content/knowledge-creation/knowledge-synthesis/formulating-the-question-eligibility-criteria-and-protocol/]
• 20. Protocol Writing in Clinical Research - PMC [https://pmc.ncbi.nlm.nih.gov/articles/PMC5198475/]
• 21. Systematic Reviews: FAQs - GalterGuides [https://libguides.galter.northwestern.edu/c.php?g=517817&p=10866177]
• 22. Screening Tools - Systematic Reviews/Evidence Synthesis [https://libguides.adelphi.edu/Systematic_Reviews/screening]
• 23. : Refining the Research Question Troubleshooting Systematic Reviews [https://www.mlanet.org/courses/troubleshooting-systematic-reviews-refining-the-research-question/]
• 24. .dml.georgetown.edu/systematicreviews guides [https://guides.dml.georgetown.edu/systematicreviews]
• 25. High-throughput screening paradigms in ecotoxicity testing [https://www.sciencedirect.com/science/article/abs/pii/S0045653522024225]
• 26. Highthroughput Ecotoxicology (HITEC) - Helmholtz-Centre ... [https://www.ufz.de/index.php?en=50906]
• 27. A case study in ecosystem condition indicators [https://www.sciencedirect.com/science/article/pii/S1574954124003613]
• 28. Software Tools to Manage Your Review - Systematic Reviews [https://prattlibrary.cchmc.org/systematicreviews/software]
• 29. Systematic Review Toolbox: Manuscript Development [https://hslib.jabsom.hawaii.edu/systematicreview/manuscriptdevelopment]
• 30. Steps of a Systematic Review - Research Guides [https://lib.guides.umd.edu/SR/steps]
• 31. Addressing challenges with systematic review teams through ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC8608185/]
• 32. Screening and Selection - Systematic Reviews [https://researchguides.library.syr.edu/c.php?g=1167626&p=8838163]
• 33. ECOTOXr: An R package for reproducible and transparent ... [https://www.sciencedirect.com/science/article/pii/S0045653524019751]
• 34. A primer on systematic reviews in toxicology - PMC [https://pmc.ncbi.nlm.nih.gov/articles/PMC5489636/]
• 35. How to troubleshoot automation problems in the lab [https://biosero.com/blog/7-ways-to-troubleshoot-automation-problems-in-the-lab/]
• 36. 7. Managing & Documenting Results - Systematic Reviews [https://libguides.ucmerced.edu/systematic-reviews/managing-results]
• 37. A comprehensive guide to conduct a systematic review and ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12366998/]
• 38. Global Distribution, Ecotoxicity, and Treatment Technologies ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12389987/]
• 39. PRISMA statement [https://www.prisma-statement.org/]
• 40. Review Aquatic ecotoxicology of anticancer drugs [https://www.sciencedirect.com/science/article/abs/pii/S0048969721046738]
• 41. TOC-LCSN Maintenance & Troubleshooting [https://www.ssi.shimadzu.com/service-support/virtual-advisor/toc-l/maintenance-and-troubleshooting/toc-lcsn/index.html]
• 42. Review and troubleshoot record import errors [https://knowledge.hubspot.com/import-and-export/troubleshoot-import-errors]
• 43. Resolve data import error messages [https://learn.microsoft.com/en-us/industry/sustainability/sustainability-manager-import-data-error-messages]
• 44. 5 Common Data Import Errors and How to Fix Them [https://www.dromo.io/blog/common-data-import-errors-and-how-to-fix-them]
• 45. Troubleshooting Data Import Errors - Dynamics 365 Community [https://community.dynamics.com/blogs/post/?postid=378e1101-df3f-4664-ba92-cfbcee83b71f]
• 46. Conflict Resolution in Research: Turning Friction into ... [https://www.srainternational.org/blogs/srai-news/2025/11/13/conflict-resolution-friction-into-collaboration]
• 47. The Future of Conflict Management: Techniques for 2025 [https://cpdonline.co.uk/knowledge-base/business/future-conflict-management/]
• 48. AI in Team Communication: Trends, Tools, and Techniques ... [https://superagi.com/ai-in-team-communication-trends-tools-and-techniques-for-enhanced-collaboration-in-2025/]
• 49. NAMs Training: Getting Started [https://www.epa.gov/chemical-research/nams-training-getting-started]
• 50. Research Screener: a machine learning tool to semi-automate ... [https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-021-01635-3]
• 51. Do it faster with PICOS: Generative AI-Assisted systematic ... [https://www.sciencedirect.com/science/article/abs/pii/S1532046425000899]
• 52. Establishment of an Integrated Model for Predicting ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12606639/]
• 53. Using artificial intelligence tools for data quality evaluation ... [https://www.sciencedirect.com/science/article/pii/S0160412025000923]
• 54. The Impact of Systematic Review Automation Tools on ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC8204237/]
• 55. Systematic Review FAQs | Harrell Health Sciences Library [https://hershey.libraries.psu.edu/services/systematic-review/faqs]
• 56. Software tools for literature screening in systematic reviews ... [https://pubmed.ncbi.nlm.nih.gov/31113000/]
• 57. Tools - Systematic Reviews & Evidence Synthesis Methods [https://guides.lib.utexas.edu/systematicreviews/tools]
• 58. The Ultimate Guide to Fine-Tuning LLMs from Basics ... [https://arxiv.org/html/2408.13296v1]
• 59. Fine-tuning, Retraining, and Beyond: Advancing with ... [https://www.analyticsvidhya.com/blog/2023/10/advancing-with-custom-llms/]
• 60. Efficient Fine-Tuning: How to Customize LLMs Without ... [https://medium.com/@a_farag/efficient-fine-tuning-how-to-customize-llms-without-breaking-the-bank-f389fab3fd79]
• 61. Customizing and fine-tuning LLMs: What you need to know [https://github.blog/ai-and-ml/llms/customizing-and-fine-tuning-llms-what-you-need-to-know/]
• 62. Software Comparison - Systematic Review Toolbox [https://hslib.jabsom.hawaii.edu/systematicreview/softwarecomparison]
• 63. 5 software tools to support your systematic review processes [https://www.ifis.org/en/research-skills-blog/software-tools-to-support-your-systematic-review-processes]
• 64. How to Train an LLM Using Fine-Tuning? Best Practices for ... [https://botscrew.com/blog/how-to-train-an-llm-using-fine-tuning/]
• 65. LibGuides: Systematic Reviews: 5. Screen Results [https://guides.mclibrary.duke.edu/sysreview/screen]
• 66. Automated medical literature screening using artificial ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC9277646/]