Debug your tasks
Sometimes things don't work out right the first time and you may need to debug your application.
- Get debug information of a task
iexec task debug <taskid> --chain bellecour
It allows anyone to know on-chain and off-chain statuses of the task.
- Get debug logs of a task
iexec task debug <taskid> --logs --chain bellecour
It allows the requester to retrieve application logs produced by workers.
During its execution, a task transitions between different off-chain statuses. Those statuses let you track how a task progresses when it's being executed and makes it easier for you to debug if the execution fails. The transitions between those statuses are as follows.
Please note that, for the sake of simplicity, transitions to the
FINAL_DEADLINE_REACHED status have not been pictured. In fact, all statuses except final statuses (FAILED and COMPLETED) can lead to this FINAL_DEADLINE_REACHED status.As a reminder, tasks have a max execution time, defined by their category. Their final deadlines are defined as follows:
deal start time + max execution time. When a task update is triggered on a Scheduler for a non-completed non-failed task while its final deadline is met, then this task status transitions to FINAL_DEADLINE_REACHED.Below the description of each status:
Task status | Description |
|---|---|
RECEIVED | The Scheduler has detected the deal |
INITIALIZING | The Scheduler is trying to set the task on-chain status to INITIALIZED |
INITIALIZED | The task on-chain status has been correctly set to INITIALIZED |
RUNNING | At least one Worker has started to work on the task |
CONSENSUS_REACHED | The consensus has been reached |
AT_LEAST_ONE_REVEALED | At least one Worker revealed its result |
RESULT_UPLOADING | The selected Worker is uploading its result |
RESULT_UPLOADED | The result has been uploaded |
FINALIZING | The Scheduler is trying to set the task on-chain status to FINALIZED |
FINALIZED | The task on-chain status has been correctly set to FINALIZED |
However, things sometimes don't work as expected. In that case, failure statuses help to understand what went wrong:
Task status | Description |
|---|---|
INITIALIZE_FAILED | Task on-chain initialization failed |
RUNNING_FAILED | All Workers have failed to run this TEE task |
CONTRIBUTION_TIMEOUT | Contribution deadline has been reached before any contribution has been sent |
RESULT_UPLOAD_TIMEOUT | The final deadline has been reached while the result was being uploaded |
FINALIZE_FAILED | Task on-chain finalization failed |
FINAL_DEADLINE_REACHED | The final deadline has been reached |
FAILED | Final status for any previous failure |
One task bought by a requester will result in one off-chain task with one or more replicates depending on the level of trust set by the requester. For a given task, each worker involved in the computation will have its own replicate containing the description of the task to compute. The whole computation of a replicate is made of several stages. Each stage completed by a worker will result in an update of its replicate status.
The links between a task to its replicates can be represented as follows:
task1
├── replicate1 (workerX)
├── replicate2 (workerY)
└── replicate3 (workerZ)
A replicate status workflow can follow two different flows:
- 1.Usual flow (default): the task is replicated on a number of workers, depending on required trust. The Scheduler has to notify the workers when the consensus is reached. It should also finalize the task on-chain.
- 2.Optimized flow (under conditions): the task is finalized by the only worker that has worked on it. It makes the task execution faster and cheaper. However, some limitations apply:
- 1.The worker has to be trustworthy. To achieve this point, only TEE tasks are eligible to this workflow.
- 2.Callback mode is currently unsupported. iExec strives to remove this limitation.
See the following flowchart for details on their transitions.
Please note that all failed status - pictured in red in the above diagram - finally lead to
ABORTED. For the sake of simplicity, this final status has not been represented here.While the task holds a meta status, each replicate has its own status which can be one of these:
Replicate status | Description |
|---|---|
CREATED | A new replicate is assigned to a worker just after it asked for more work |
STARTING | The worker starts preflight checks to confirm it can work on this replicate |
STARTED | The worker confirms it is going to work on this replicate |
START_FAILED | The preflight checks have failed. The worker will NOT work on this replicate |
APP_DOWNLOADING | The worker is downloading the application |
APP_DOWNLOADED | The download of the application is completed |
APP_DOWNLOAD_FAILED | The download of the application failed |
DATA_DOWNLOADING | The worker is downloading the dataset |
DATA_DOWNLOADED | The download of the dataset is completed |
DATA_DOWNLOAD_FAILED | The download of the dataset failed |
COMPUTING | The worker is computing the task |
COMPUTED | The computation is completed |
COMPUTE_FAILED | The computation failed |
CONTRIBUTING | The worker sent the "contribute(..)" transaction (result digest) on chain |
CONTRIBUTE_FAILED | The contribute transaction failed |
CONTRIBUTED | The worker has contributed on chain |
REVEALING | The worker sent the "reveal(..)" transaction (proof that he is the owner of the result digest) |
REVEALED | The worker has revealed the proof on chain |
REVEAL_FAILED | The reveal transaction failed |
RESULT_UPLOAD_REQUESTED | The worker has been requested to upload the result to a remote filesystem |
RESULT_UPLOADING | The worker is uploading the result |
RESULT_UPLOAD_FAILED | The upload of the result failed |
RESULT_UPLOADED | The result has been uploaded to IPFS over the iExec Result Proxy (standard or TEE tasks) or to Dropbox (TEE only), dependending on the deal parameters |
CONTRIBUTE_AND_FINALIZE_ONGOING | The worker sent the "contributeAndFinalize(...)" transaction on chain |
CONTRIBUTE_AND_FINALIZE_DONE | The worker has contributed and finalized the task. The latter is now considered as completed on-chain |
CONTRIBUTE_AND_FINALIZE_FAILED | The contributeAndFinalize transaction failed |
COMPLETING | The task is finalized, the worker will purge data related to its replicate |
COMPLETED | The whole task is completed meaning the task is finalized. The worker has been rewarded if it is part of the consensus |
COMPLETE_FAILED | The worker failed to clean the local replicate resources after the task is finalized |
FAILED | The worker failed to participate to the task |
ABORTED | The scheduler asked the worker to stop working on this replicate while the latter was still working on it |
RECOVERING | The worker has been stopped, it is starting back from where it stopped |
WORKER_LOST | The worker didn't ping the iexec-core scheduler for a while. It is considered as out for this task |
When a worker fails to complete a replicate, it returns a failure cause. This cause is helpful to understand what went wrong.
Replicate failure cause | Description |
|---|---|
REVEAL_TIMEOUT | The worker took too long to reveal its proof (more than 2 periods after the consensus) |
A replicate can fail with the following causes:
Common failures
Replicate failure cause | Replicate status | Description |
|---|---|---|
CHAIN_UNREACHABLE | STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING | The task model could not be fetched from the blockchain |
STAKE_TOO_LOW | STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING | Worker deposit is too low |
TASK_NOT_ACTIVE | STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING | On-chain task status is not ACTIVE |
CONTRIBUTION_TIMEOUT | STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING | Contribution deadline has already been reached |
CONTRIBUTION_ALREADY_SET | STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING | The worker has already contributed |
WORKERPOOL_AUTHORIZATION_NOT_FOUND | STARTING, APP_DOWNLOADING, DATA_DOWNLOADING, COMPUTING, CONTRIBUTING | The authorization to contribute to the task is missing |
APP_IMAGE_DOWNLOAD_FAILED | APP_DOWNLOADING | The download of the application image failed |
APP_NOT_FOUND_LOCALLY | COMPUTING | The application image could not be found on the worker |
APP_COMPUTE_FAILED | COMPUTING | The application execution failed |
POST_COMPUTE_COMPUTED_FILE_NOT_FOUND | COMPUTING | The computed.json file could not be found |
POST_COMPUTE_RESULT_DIGEST_COMPUTATION_FAILED | COMPUTING | The result digest could not be computed from the computed.json file |
POST_COMPUTE_TOO_LONG_RESULT_FILE_NAME | COMPUTING | One or more of the result files name exceed the limit of 31 characters |
POST_COMPUTE_OUT_FOLDER_ZIP_FAILED | COMPUTING | post-compute failed to zip the output folder resulting from the computation |
POST_COMPUTE_SEND_COMPUTED_FILE_FAILED | COMPUTING | Failed to post computed.json to worker |
OUT_OF_GAS | CONTRIBUTING, REVEALING | The worker needs some ETH, please refill its wallet |
DETERMINISM_HASH_NOT_FOUND | CONTRIBUTING, REVEALING | The result digest could not be read from the computed.json file |
CHAIN_RECEIPT_NOT_VALID | CONTRIBUTING, REVEALING | The transaction failed on the blockchain |
CONSENSUS_BLOCK_MISSING | REVEALING | The consensus block number was not part of the PLEASE_REVEAL notification |
BLOCK_NOT_REACHED | REVEALING | The worker has not reached the consensus block |
CANNOT_REVEAL | REVEALING | One of the mandatory condition was not met. Reveal cannot happen |
RESULT_LINK_MISSING | UPLOADING | No result link has been provided by the worker |
Specific failures for standard tasks
Replicate failure cause | Replicate status | Description |
|---|---|---|
TASK_DESCRIPTION_INVALID | STARTING | The task description contains inconsistencies and cannot be executed |
DATASET_FILE_DOWNLOAD_FAILED | DATA_DOWNLOADING | Dataset download failed |
DATASET_FILE_BAD_CHECKSUM | DATA_DOWNLOADING | Downloaded dataset checksum does not match on-chain provided checksum |
INPUT_FILES_DOWNLOAD_FAILED | DATA_DOWNLOADING | At least one input file could not be downloaded |
Specific failures for TEE tasks
Replicate failure cause | Replicate status | Description |
|---|---|---|
TEE_NOT_SUPPORTED | STARTING | The current worker does not support TEE tasks. It may not be well configured or not compatible at all |
UNKNOWN_SMS | STARTING | SMS URL could not be resolved for this task |
GET_TEE_SERVICES_CONFIGURATION_FAILED | STARTING | Failed to fetch TEE task configuration properties from SMS |
TEE_PREPARATION_FAILED | COMPUTING | TEE task preparation step could not be completed, task cannot be executed by the worker |
TEE_SESSION_GENERATION_INVALID_AUTHORIZATION | COMPUTING | The worker could not authenticate to the SMS |
TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_EMPTY_PARAMS_UNAUTHORIZED | COMPUTING | The parameters provided by the worker do not allow the TEE session generation |
TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_NO_MATCH_ONCHAIN_TYPE | COMPUTING | The provided task ID does not match with a TEE task on-chain |
TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_GET_CHAIN_TASK_FAILED | COMPUTING | The associated task could not be retrieved on-chain |
TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_TASK_NOT_ACTIVE | COMPUTING | The task is not active on chain |
TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_GET_CHAIN_DEAL_FAILED | COMPUTING | The associated deal could not be retrieved on-chain |
TEE_SESSION_GENERATION_EXECUTION_NOT_AUTHORIZED_INVALID_SIGNATURE | COMPUTING | The provided authorization has not been signed by the correct workerpool owner |
TEE_SESSION_GENERATION_PRE_COMPUTE_GET_DATASET_SECRET_FAILED | COMPUTING | Dataset secret could not be fetched from the SMS database |
TEE_SESSION_GENERATION_APP_COMPUTE_NO_ENCLAVE_CONFIG | COMPUTING | Application enclave configuration could not be read from on-chain app description |
TEE_SESSION_GENERATION_APP_COMPUTE_INVALID_ENCLAVE_CONFIG | COMPUTING | Application enclave configuration could not be validated. At least one item between the entrypoint, the heap size or the fingerprint of the app enclave description is not correct |
TEE_SESSION_GENERATION_POST_COMPUTE_GET_ENCRYPTION_TOKENS_FAILED_EMPTY_BENEFICIARY_KEY | COMPUTING | Beneficiary result encryption key could not be fetched from the SMS database |
TEE_SESSION_GENERATION_POST_COMPUTE_GET_STORAGE_TOKENS_FAILED | COMPUTING | Results storage (IPFS or Dropbox) token could not be fetched from the SMS database |
TEE_SESSION_GENERATION_POST_COMPUTE_GET_SIGNATURE_TOKENS_FAILED_EMPTY_WORKER_ADDRESS | COMPUTING | The worker address is missing to compute the enclave signature |
TEE_SESSION_GENERATION_POST_COMPUTE_GET_SIGNATURE_TOKENS_FAILED_EMPTY_PUBLIC_ENCLAVE_CHALLENGE | COMPUTING | The enclave challenge is missing for the enclave signature computation |
TEE_SESSION_GENERATION_POST_COMPUTE_GET_SIGNATURE_TOKENS_FAILED_EMPTY_TEE_CHALLENGE | COMPUTING | The TEE challenge is missing for the enclave signature computation |
TEE_SESSION_GENERATION_POST_COMPUTE_GET_SIGNATURE_TOKENS_FAILED_EMPTY_TEE_CREDENTIALS | COMPUTING | The TEE credentials are missing for the enclave signature computation |
TEE_SESSION_GENERATION_SECURE_SESSION_STORAGE_CALL_FAILED | COMPUTING | The TEE session could not be posted on the secure session storage |
TEE_SESSION_GENERATION_SECURE_SESSION_GENERATION_FAILED | COMPUTING | The TEE session could not be generated but the SMS can't provide the reason |
TEE_SESSION_GENERATION_SECURE_SESSION_NO_TEE_PROVIDER | COMPUTING | No TEE provider has been found on the task description |
TEE_SESSION_GENERATION_GET_TASK_DESCRIPTION_FAILED | COMPUTING | The task description could not be read on-chain |
TEE_SESSION_GENERATION_NO_SESSION_REQUEST | COMPUTING | The TEE session request was empty |
TEE_SESSION_GENERATION_NO_TASK_DESCRIPTION | COMPUTING | The task description has not been found in the TEE session request |
TEE_SESSION_GENERATION_UNKNOWN_ISSUE | COMPUTING | The TEE session could not be generated but the SMS hasn't provided any reason |
PRE_COMPUTE_INVALID_ENCLAVE_CONFIGURATION | COMPUTING | Application enclave configuration is invalid |
PRE_COMPUTE_INVALID_ENCLAVE_HEAP_CONFIGURATION | COMPUTING | Application requested HEAP size is bigger than maximum allocatable memory |
PRE_COMPUTE_IMAGE_MISSING | COMPUTING | The pre-compute image could not be found on the worker |
PRE_COMPUTE_TASK_ID_MISSING | COMPUTING | The IEXEC_TASK_ID environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_EXIT_REPORTING_FAILED | COMPUTING | The pre-compute container failed to post its failure cause to the worker |
PRE_COMPUTE_OUTPUT_PATH_MISSING | COMPUTING | The IEXEC_PRE_COMPUTE_OUT environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_IS_DATASET_REQUIRED_MISSING | COMPUTING | The IS_DATASET_REQUIRED environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_DATASET_URL_MISSING | COMPUTING | The IEXEC_DATASET_URL environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_DATASET_KEY_MISSING | COMPUTING | The IEXEC_DATASET_KEY environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_DATASET_CHECKSUM_MISSING | COMPUTING | The IEXEC_DATASET_CHECKSUM environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_DATASET_FILENAME_MISSING | COMPUTING | The IEXEC_DATASET_FILENAME environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_INPUT_FILES_NUMBER_MISSING | COMPUTING | The IEXEC_INPUT_FILES_NUMBER environment variable could not be resolved from the pre-compute container |
PRE_COMPUTE_AT_LEAST_ONE_INPUT_FILE_URL_MISSING | COMPUTING | At least one environment variable starting with IEXEC_INPUT_FILE_URL_ prefix could not be resolve from the pre-compute container |
PRE_COMPUTE_OUTPUT_FOLDER_NOT_FOUND | COMPUTING | The folder to store the dataset or input files could not be found |
PRE_COMPUTE_DATASET_DOWNLOAD_FAILED | COMPUTING | The pre-compute failed to download the dataset |
PRE_COMPUTE_INVALID_DATASET_CHECKSUM | COMPUTING | The downloaded dataset checksum does not match the expected one |
PRE_COMPUTE_DATASET_DECRYPTION_FAILED | COMPUTING | The downloaded dataset decryption failed |
PRE_COMPUTE_SAVING_PLAIN_DATASET_FAILED | COMPUTING | The decrypted dataset could not be saved in the enclave |
PRE_COMPUTE_INPUT_FILE_DOWNLOAD_FAILED | COMPUTING | The pre-commpute failed to download at least one input file |
PRE_COMPUTE_TIMEOUT | COMPUTING | The pre-compute did not complete on time |
PRE_COMPUTE_FAILED_UNKNOWN_ISSUE | COMPUTING | The pre-compute failed for an unknown cause |
POST_COMPUTE_IMAGE_MISSING | COMPUTING | The post-compute image could not be found on the worker |
POST_COMPUTE_TASK_ID_MISSING | COMPUTING | The RESULT_TASK_ID environment variable could not be resolved from the post-compute container |
POST_COMPUTE_EXIT_REPORTING_FAILED | COMPUTING | The post-compute container failed to post its failure cause to the worker |
POST_COMPUTE_WORKER_ADDRESS_MISSING | COMPUTING | The RESULT_SIGN_WORKER_ADDRESS environment variable could not be resolved from the post-compute container |
POST_COMPUTE_TEE_CHALLENGE_PRIVATE_KEY_MISSING | COMPUTING | The RESULT_SIGN_TEE_CHALLENGE_PRIVATE_KEY environment variable could not be resolved from the post-compute container |
POST_COMPUTE_ENCRYPTION_PUBLIC_KEY_MISSING | COMPUTING | The RESULT_ENCRYPTION_PUBLIC_KEY environment variable could not be resolved from the post-compute container |
POST_COMPUTE_STORAGE_TOKEN_MISSING | COMPUTING | The RESULT_STORAGE_TOKEN environment variable could not be resolved from the post-compute container |
POST_COMPUTE_ENCRYPTION_FAILED | COMPUTING | Results encryption failed in post-compute container |
POST_COMPUTE_RESULT_FILE_NOT_FOUND | COMPUTING | Local result file to upload could not be found |
POST_COMPUTE_DROPBOX_UPLOAD_FAILED | COMPUTING | Upload to DROPBOX failed |
POST_COMPUTE_IPFS_UPLOAD_FAILED | COMPUTING | Upload to IPFS failed |
POST_COMPUTE_INVALID_TEE_SIGNATURE | COMPUTING | The provided signed TEE challenge is invalid |
POST_COMPUTE_TIMEOUT | COMPUTING | The post-compute did not complete on time |
POST_COMPUTE_FAILED_UNKOWN_ISSUE | COMPUTING | The post-compute failed for an unknown cause |
ENCLAVE_SIGNATURE_NOT_FOUND | CONTRIBUTING | The enclave signature could not be found for the result. The worker will not contribute |
Last modified 1mo ago