nvme: Fail passthrough commands right away in failed state.

When the drive is failed, we can't send passthrough commands to the
card, so fail them right away. Rearrange the comments to reflect the
current failure paths in the driver.

Sponsored by:		Netflix
Differential Revision:	https://reviews.freebsd.org/D46048

sys/dev/nvme/nvme_qpair.c

@@ -1267,41 +1267,31 @@ _nvme_qpair_submit_request(struct nvme_qpair *qpair, struct nvme_request *req)
 	tr = TAILQ_FIRST(&qpair->free_tr);
 	req->qpair = qpair;
 
-	if (tr == NULL || qpair->recovery_state != RECOVERY_NONE) {
+	/*
-		/*
+	 * The controller has failed, so fail the request. Note, that this races
-		 * No tracker is available, or the qpair is disabled due to an
+	 * the recovery / timeout code. Since we hold the qpair lock, we know
-		 * in-progress controller-level reset. If we lose the race with
+	 * it's safe to fail directly. is_failed is set when we fail the controller.
-		 * recovery_state, then we may add an extra request to the queue
+	 * It is only ever reset in the ioctl reset controller path, which is safe
-		 * which will be resubmitted later.  We only set recovery_state
+	 * to race (for failed controllers, we make no guarantees about bringing
-		 * to NONE with qpair->lock also held, so if we observe that the
+	 * it out of failed state relative to other commands).
-		 * state is not NONE, we know it can't transition to NONE below
+	 */
-		 * when we've submitted the request to hardware.
+	if (qpair->ctrlr->is_failed) {
-		 *
+		nvme_qpair_manual_complete_request(qpair, req,
-		 * Also, as part of the failure process, we set recovery_state
+		    NVME_SCT_GENERIC, NVME_SC_ABORTED_BY_REQUEST);
-		 * to RECOVERY_WAITING, so we check here to see if we've failed
+		return;
-		 * the controller. We set it before we call the qpair_fail
+	}
-		 * functions, which take out the lock lock before messing with
-		 * queued_req. Since we hold that lock, we know it's safe to
-		 * either fail directly, or queue the failure should is_failed
-		 * be stale. If we lose the race reading is_failed, then
-		 * nvme_qpair_fail will fail the queued request.
-		 */
 
-		if (qpair->ctrlr->is_failed) {
+	/*
-			/*
+	 * No tracker is available, or the qpair is disabled due to an
-			 * The controller has failed, so fail the request.
+	 * in-progress controller-level reset. If we lose the race with
-			 */
+	 * recovery_state, then we may add an extra request to the queue which
-			nvme_qpair_manual_complete_request(qpair, req,
+	 * will be resubmitted later.  We only set recovery_state to NONE with
-			    NVME_SCT_GENERIC, NVME_SC_ABORTED_BY_REQUEST);
+	 * qpair->lock also held, so if we observe that the state is not NONE,
-		} else {
+	 * we know it won't transition back to NONE without retrying queued
-			/*
+	 * request.
-			 * Put the request on the qpair's request queue to be
+	 */
-			 *  processed when a tracker frees up via a command
+	if (tr == NULL || qpair->recovery_state != RECOVERY_NONE) {
-			 *  completion or when the controller reset is
+		STAILQ_INSERT_TAIL(&qpair->queued_req, req, stailq);
-			 *  completed.
-			 */
-			STAILQ_INSERT_TAIL(&qpair->queued_req, req, stailq);
-		}
 		return;
 	}