If there are errors with ingests, there are a few steps to take to research and resolve the issue.

Troubleshooting questions for Ingests:

1. Are all Corpora configured with a correct annis_corpus_name to ingest data from ANNIS?

2. How long has it been since the ingest was created? An ingest needs at most about 30 seconds per text involved in the ingest.

3. Is the ANNIS server responsive? If the ANNIS server is unresponsive or unreachable by the Ingest, the Ingest will fail.

4. Was another ingest started while a previous ingest was completing? If so, both ingests will fail.

5. Has a previous ingest failed recently. If so, it might be necessary to restart the URN resolver ingest background process.

Responding to issues:

If a ingest fails for any reason and prevents new ingests from being created, there are a few things to check on the server.

1. Restart the coptic celery background ingest process. If there is a stalled ingest for any reason, one potential solution is to restart the background process that manages ingests.

To do this, follow these steps:
1.a. Connect to the server via SSH
1.b. Review the ingest log file at /home/ubuntu/celery/coptic_django/ logs/celery-worker.log and ensure no ingest is running that is actively adding updates to the log file.
1.c. Type the command “sudo supervisorctl” to enter the supervisor control.
1.d. Type “restart coptic-celery” to restart the coptic celery ingest background process. This may require a few moments.

2. Kill the firefox and/or virtual frame buffer (Xvfb) processes that may be remaining from a previous ingest. These processes may be taking up additional memory.

2.a. Connect to the server via SSH
2.b. Review the ingest log file at /home/ubuntu/celery/coptic_django/ logs/celery-worker.log and ensure no ingest is running that is actively adding updates to the log file.
2.c. Type the command “top” and push enter.  This will bring up the top panel of running processes on the server.
2.d. Hold shift and press “m” to sort the processes by memory.  
2.e. If in the “Command” column, there is listed “firefox” or “Xvfb”, these processes need to be stopped to free memory resources on the server.
2.f. Type “pkill firefox” and “pkill Xvfb”.  
2.g. Repeat steps 2.c - 2.e and ensure both processes have ended and no longer appear in the top active processes monitor panel.

3. Review the log file for the error and consult a developer or administrator.

3.a. Connect to the server via SSH
3.b. Review the ingest log file at /home/ubuntu/celery/coptic_django/logs/
celery-worker.log and look for the most recent error (the error furthest toward the bottom of the file).
3.c. Send the error report and other information to a developer or systems administrator for resolution.