diff options
Diffstat (limited to 'Documentation/scsi/scsi_eh.txt')
| -rw-r--r-- | Documentation/scsi/scsi_eh.txt | 475 |
1 files changed, 0 insertions, 475 deletions
diff --git a/Documentation/scsi/scsi_eh.txt b/Documentation/scsi/scsi_eh.txt deleted file mode 100644 index 1b7436932a2b..000000000000 --- a/Documentation/scsi/scsi_eh.txt +++ /dev/null @@ -1,475 +0,0 @@ - -SCSI EH -====================================== - - This document describes SCSI midlayer error handling infrastructure. -Please refer to Documentation/scsi/scsi_mid_low_api.txt for more -information regarding SCSI midlayer. - -TABLE OF CONTENTS - -[1] How SCSI commands travel through the midlayer and to EH - [1-1] struct scsi_cmnd - [1-2] How do scmd's get completed? - [1-2-1] Completing a scmd w/ scsi_done - [1-2-2] Completing a scmd w/ timeout - [1-3] How EH takes over -[2] How SCSI EH works - [2-1] EH through fine-grained callbacks - [2-1-1] Overview - [2-1-2] Flow of scmds through EH - [2-1-3] Flow of control - [2-2] EH through transportt->eh_strategy_handler() - [2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions - [2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions - [2-2-3] Things to consider - - -[1] How SCSI commands travel through the midlayer and to EH - -[1-1] struct scsi_cmnd - - Each SCSI command is represented with struct scsi_cmnd (== scmd). A -scmd has two list_head's to link itself into lists. The two are -scmd->list and scmd->eh_entry. The former is used for free list or -per-device allocated scmd list and not of much interest to this EH -discussion. The latter is used for completion and EH lists and unless -otherwise stated scmds are always linked using scmd->eh_entry in this -discussion. - - -[1-2] How do scmd's get completed? - - Once LLDD gets hold of a scmd, either the LLDD will complete the -command by calling scsi_done callback passed from midlayer when -invoking hostt->queuecommand() or the block layer will time it out. - - -[1-2-1] Completing a scmd w/ scsi_done - - For all non-EH commands, scsi_done() is the completion callback. It -just calls blk_complete_request() to delete the block layer timer and -raise SCSI_SOFTIRQ - - SCSI_SOFTIRQ handler scsi_softirq calls scsi_decide_disposition() to -determine what to do with the command. scsi_decide_disposition() -looks at the scmd->result value and sense data to determine what to do -with the command. - - - SUCCESS - scsi_finish_command() is invoked for the command. The - function does some maintenance chores and then calls - scsi_io_completion() to finish the I/O. - scsi_io_completion() then notifies the block layer on - the completed request by calling blk_end_request and - friends or figures out what to do with the remainder - of the data in case of an error. - - - NEEDS_RETRY - - ADD_TO_MLQUEUE - scmd is requeued to blk queue. - - - otherwise - scsi_eh_scmd_add(scmd) is invoked for the command. See - [1-3] for details of this function. - - -[1-2-2] Completing a scmd w/ timeout - - The timeout handler is scsi_times_out(). When a timeout occurs, this -function - - 1. invokes optional hostt->eh_timed_out() callback. Return value can - be one of - - - BLK_EH_RESET_TIMER - This indicates that more time is required to finish the - command. Timer is restarted. This action is counted as a - retry and only allowed scmd->allowed + 1(!) times. Once the - limit is reached, action for BLK_EH_DONE is taken instead. - - - BLK_EH_DONE - eh_timed_out() callback did not handle the command. - Step #2 is taken. - - 2. scsi_abort_command() is invoked to schedule an asynchrous abort. - Asynchronous abort are not invoked for commands which the - SCSI_EH_ABORT_SCHEDULED flag is set (this indicates that the command - already had been aborted once, and this is a retry which failed), - or when the EH deadline is expired. In these case Step #3 is taken. - - 3. scsi_eh_scmd_add(scmd, SCSI_EH_CANCEL_CMD) is invoked for the - command. See [1-4] for more information. - -[1-3] Asynchronous command aborts - - After a timeout occurs a command abort is scheduled from - scsi_abort_command(). If the abort is successful the command - will either be retried (if the number of retries is not exhausted) - or terminated with DID_TIME_OUT. - Otherwise scsi_eh_scmd_add() is invoked for the command. - See [1-4] for more information. - -[1-4] How EH takes over - - scmds enter EH via scsi_eh_scmd_add(), which does the following. - - 1. Links scmd->eh_entry to shost->eh_cmd_q - - 2. Sets SHOST_RECOVERY bit in shost->shost_state - - 3. Increments shost->host_failed - - 4. Wakes up SCSI EH thread if shost->host_busy == shost->host_failed - - As can be seen above, once any scmd is added to shost->eh_cmd_q, -SHOST_RECOVERY shost_state bit is turned on. This prevents any new -scmd to be issued from blk queue to the host; eventually, all scmds on -the host either complete normally, fail and get added to eh_cmd_q, or -time out and get added to shost->eh_cmd_q. - - If all scmds either complete or fail, the number of in-flight scmds -becomes equal to the number of failed scmds - i.e. shost->host_busy == -shost->host_failed. This wakes up SCSI EH thread. So, once woken up, -SCSI EH thread can expect that all in-flight commands have failed and -are linked on shost->eh_cmd_q. - - Note that this does not mean lower layers are quiescent. If a LLDD -completed a scmd with error status, the LLDD and lower layers are -assumed to forget about the scmd at that point. However, if a scmd -has timed out, unless hostt->eh_timed_out() made lower layers forget -about the scmd, which currently no LLDD does, the command is still -active as long as lower layers are concerned and completion could -occur at any time. Of course, all such completions are ignored as the -timer has already expired. - - We'll talk about how SCSI EH takes actions to abort - make LLDD -forget about - timed out scmds later. - - -[2] How SCSI EH works - - LLDD's can implement SCSI EH actions in one of the following two -ways. - - - Fine-grained EH callbacks - LLDD can implement fine-grained EH callbacks and let SCSI - midlayer drive error handling and call appropriate callbacks. - This will be discussed further in [2-1]. - - - eh_strategy_handler() callback - This is one big callback which should perform whole error - handling. As such, it should do all chores the SCSI midlayer - performs during recovery. This will be discussed in [2-2]. - - Once recovery is complete, SCSI EH resumes normal operation by -calling scsi_restart_operations(), which - - 1. Checks if door locking is needed and locks door. - - 2. Clears SHOST_RECOVERY shost_state bit - - 3. Wakes up waiters on shost->host_wait. This occurs if someone - calls scsi_block_when_processing_errors() on the host. - (*QUESTION* why is it needed? All operations will be blocked - anyway after it reaches blk queue.) - - 4. Kicks queues in all devices on the host in the asses - - -[2-1] EH through fine-grained callbacks - -[2-1-1] Overview - - If eh_strategy_handler() is not present, SCSI midlayer takes charge -of driving error handling. EH's goals are two - make LLDD, host and -device forget about timed out scmds and make them ready for new -commands. A scmd is said to be recovered if the scmd is forgotten by -lower layers and lower layers are ready to process or fail the scmd -again. - - To achieve these goals, EH performs recovery actions with increasing -severity. Some actions are performed by issuing SCSI commands and -others are performed by invoking one of the following fine-grained -hostt EH callbacks. Callbacks may be omitted and omitted ones are -considered to fail always. - -int (* eh_abort_handler)(struct scsi_cmnd *); -int (* eh_device_reset_handler)(struct scsi_cmnd *); -int (* eh_bus_reset_handler)(struct scsi_cmnd *); -int (* eh_host_reset_handler)(struct scsi_cmnd *); - - Higher-severity actions are taken only when lower-severity actions -cannot recover some of failed scmds. Also, note that failure of the -highest-severity action means EH failure and results in offlining of -all unrecovered devices. - - During recovery, the following rules are followed - - - Recovery actions are performed on failed scmds on the to do list, - eh_work_q. If a recovery action succeeds for a scmd, recovered - scmds are removed from eh_work_q. - - Note that single recovery action on a scmd can recover multiple - scmds. e.g. resetting a device recovers all failed scmds on the - device. - - - Higher severity actions are taken iff eh_work_q is not empty after - lower severity actions are complete. - - - EH reuses failed scmds to issue commands for recovery. For - timed-out scmds, SCSI EH ensures that LLDD forgets about a scmd - before reusing it for EH commands. - - When a scmd is recovered, the scmd is moved from eh_work_q to EH -local eh_done_q using scsi_eh_finish_cmd(). After all scmds are -recovered (eh_work_q is empty), scsi_eh_flush_done_q() is invoked to -either retry or error-finish (notify upper layer of failure) recovered -scmds. - - scmds are retried iff its sdev is still online (not offlined during -EH), REQ_FAILFAST is not set and ++scmd->retries is less than -scmd->allowed. - - -[2-1-2] Flow of scmds through EH - - 1. Error completion / time out - ACTION: scsi_eh_scmd_add() is invoked for scmd - - add scmd to shost->eh_cmd_q - - set SHOST_RECOVERY - - shost->host_failed++ - LOCKING: shost->host_lock - - 2. EH starts - ACTION: move all scmds to EH's local eh_work_q. shost->eh_cmd_q - is cleared. - LOCKING: shost->host_lock (not strictly necessary, just for - consistency) - - 3. scmd recovered - ACTION: scsi_eh_finish_cmd() is invoked to EH-finish scmd - - scsi_setup_cmd_retry() - - move from local eh_work_q to local eh_done_q - LOCKING: none - CONCURRENCY: at most one thread per separate eh_work_q to - keep queue manipulation lockless - - 4. EH completes - ACTION: scsi_eh_flush_done_q() retries scmds or notifies upper - layer of failure. May be called concurrently but must have - a no more than one thread per separate eh_work_q to - manipulate the queue locklessly - - scmd is removed from eh_done_q and scmd->eh_entry is cleared - - if retry is necessary, scmd is requeued using - scsi_queue_insert() - - otherwise, scsi_finish_command() is invoked for scmd - - zero shost->host_failed - LOCKING: queue or finish function performs appropriate locking - - -[2-1-3] Flow of control - - EH through fine-grained callbacks start from scsi_unjam_host(). - -<<scsi_unjam_host>> - - 1. Lock shost->host_lock, splice_init shost->eh_cmd_q into local - eh_work_q and unlock host_lock. Note that shost->eh_cmd_q is - cleared by this action. - - 2. Invoke scsi_eh_get_sense. - - <<scsi_eh_get_sense>> - - This action is taken for each error-completed - (!SCSI_EH_CANCEL_CMD) commands without valid sense data. Most - SCSI transports/LLDDs automatically acquire sense data on - command failures (autosense). Autosense is recommended for - performance reasons and as sense information could get out of - sync between occurrence of CHECK CONDITION and this action. - - Note that if autosense is not supported, scmd->sense_buffer - contains invalid sense data when error-completing the scmd - with scsi_done(). scsi_decide_disposition() always returns - FAILED in such cases thus invoking SCSI EH. When the scmd - reaches here, sense data is acquired and - scsi_decide_disposition() is called again. - - 1. Invoke scsi_request_sense() which issues REQUEST_SENSE - command. If fails, no action. Note that taking no action - causes higher-severity recovery to be taken for the scmd. - - 2. Invoke scsi_decide_disposition() on the scmd - - - SUCCESS - scmd->retries is set to scmd->allowed preventing - scsi_eh_flush_done_q() from retrying the scmd and - scsi_eh_finish_cmd() is invoked. - - - NEEDS_RETRY - scsi_eh_finish_cmd() invoked - - - otherwise - No action. - - 3. If !list_empty(&eh_work_q), invoke scsi_eh_abort_cmds(). - - <<scsi_eh_abort_cmds>> - - This action is taken for each timed out command when - no_async_abort is enabled in the host template. - hostt->eh_abort_handler() is invoked for each scmd. The - handler returns SUCCESS if it has succeeded to make LLDD and - all related hardware forget about the scmd. - - If a timedout scmd is successfully aborted and the sdev is - either offline or ready, scsi_eh_finish_cmd() is invoked for - the scmd. Otherwise, the scmd is left in eh_work_q for - higher-severity actions. - - Note that both offline and ready status mean that the sdev is - ready to process new scmds, where processing also implies - immediate failing; thus, if a sdev is in one of the two - states, no further recovery action is needed. - - Device readiness is tested using scsi_eh_tur() which issues - TEST_UNIT_READY command. Note that the scmd must have been - aborted successfully before reusing it for TEST_UNIT_READY. - - 4. If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs() - - <<scsi_eh_ready_devs>> - - This function takes four increasingly more severe measures to - make failed sdevs ready for new commands. - - 1. Invoke scsi_eh_stu() - - <<scsi_eh_stu>> - - For each sdev which has failed scmds with valid sense data - of which scsi_check_sense()'s verdict is FAILED, - START_STOP_UNIT command is issued w/ start=1. Note that - as we explicitly choose error-completed scmds, it is known - that lower layers have forgotten about the scmd and we can - reuse it for STU. - - If STU succeeds and the sdev is either offline or ready, - all failed scmds on the sdev are EH-finished with - scsi_eh_finish_cmd(). - - *NOTE* If hostt->eh_abort_handler() isn't implemented or - failed, we may still have timed out scmds at this point - and STU doesn't make lower layers forget about those - scmds. Yet, this function EH-finish all scmds on the sdev - if STU succeeds leaving lower layers in an inconsistent - state. It seems that STU action should be taken only when - a sdev has no timed out scmd. - - 2. If !list_empty(&eh_work_q), invoke scsi_eh_bus_device_reset(). - - <<scsi_eh_bus_device_reset>> - - This action is very similar to scsi_eh_stu() except that, - instead of issuing STU, hostt->eh_device_reset_handler() - is used. Also, as we're not issuing SCSI commands and - resetting clears all scmds on the sdev, there is no need - to choose error-completed scmds. - - 3. If !list_empty(&eh_work_q), invoke scsi_eh_bus_reset() - - <<scsi_eh_bus_reset>> - - hostt->eh_bus_reset_handler() is invoked for each channel - with failed scmds. If bus reset succeeds, all failed - scmds on all ready or offline sdevs on the channel are - EH-finished. - - 4. If !list_empty(&eh_work_q), invoke scsi_eh_host_reset() - - <<scsi_eh_host_reset>> - - This is the last resort. hostt->eh_host_reset_handler() - is invoked. If host reset succeeds, all failed scmds on - all ready or offline sdevs on the host are EH-finished. - - 5. If !list_empty(&eh_work_q), invoke scsi_eh_offline_sdevs() - - <<scsi_eh_offline_sdevs>> - - Take all sdevs which still have unrecovered scmds offline - and EH-finish the scmds. - - 5. Invoke scsi_eh_flush_done_q(). - - <<scsi_eh_flush_done_q>> - - At this point all scmds are recovered (or given up) and - put on eh_done_q by scsi_eh_finish_cmd(). This function - flushes eh_done_q by either retrying or notifying upper - layer of failure of the scmds. - - -[2-2] EH through transportt->eh_strategy_handler() - - transportt->eh_strategy_handler() is invoked in the place of -scsi_unjam_host() and it is responsible for whole recovery process. -On completion, the handler should have made lower layers forget about -all failed scmds and either ready for new commands or offline. Also, -it should perform SCSI EH maintenance chores to maintain integrity of -SCSI midlayer. IOW, of the steps described in [2-1-2], all steps -except for #1 must be implemented by eh_strategy_handler(). - - -[2-2-1] Pre transportt->eh_strategy_handler() SCSI midlayer conditions - - The following conditions are true on entry to the handler. - - - Each failed scmd's eh_flags field is set appropriately. - - - Each failed scmd is linked on scmd->eh_cmd_q by scmd->eh_entry. - - - SHOST_RECOVERY is set. - - - shost->host_failed == shost->host_busy - - -[2-2-2] Post transportt->eh_strategy_handler() SCSI midlayer conditions - - The following conditions must be true on exit from the handler. - - - shost->host_failed is zero. - - - Each scmd is in such a state that scsi_setup_cmd_retry() on the - scmd doesn't make any difference. - - - shost->eh_cmd_q is cleared. - - - Each scmd->eh_entry is cleared. - - - Either scsi_queue_insert() or scsi_finish_command() is called on - each scmd. Note that the handler is free to use scmd->retries and - ->allowed to limit the number of retries. - - -[2-2-3] Things to consider - - - Know that timed out scmds are still active on lower layers. Make - lower layers forget about them before doing anything else with - those scmds. - - - For consistency, when accessing/modifying shost data structure, - grab shost->host_lock. - - - On completion, each failed sdev must have forgotten about all - active scmds. - - - On completion, each failed sdev must be ready for new commands or - offline. - - --- -Tejun Heo -htejun@gmail.com -11th September 2005 |