summaryrefslogtreecommitdiffstats
path: root/block/blk-mq.c
Commit message (Collapse)AuthorAgeFilesLines
...
* blk-mq: don't allow queue entering for a dying queueKeith Busch2014-06-061-2/+4
| | | | | | | | | If the queue is going away, don't let new allocs or queueing happen on it. Go through the normal wait process, and exit with ENODEV in that case. Signed-off-by: Keith Busch <keith.busch@intel.com> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: bump max tag depth to 10K tagsJens Axboe2014-06-061-1/+12
| | | | | | | | | | | For some scsi-mq cases, the tag map can be huge. So increase the max number of tags we support. Additionally, don't fail with EINVAL if a user requests too many tags. Warn that the tag depth has been adjusted down, and store the new value inside the tag_set passed in. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: let blk_mq_tag_to_rq() take blk_mq_tags as the main parameterJens Axboe2014-06-041-7/+12
| | | | | | | | | | We currently pass in the hardware queue, and get the tags from there. But from scsi-mq, with a shared tag space, it's a lot more convenient to pass in the blk_mq_tags instead as the hardware queue isn't always directly available. So instead of having to re-map to a given hardware queue from rq->mq_ctx, just pass in the tags structure. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: fix regression from commit 624dbe475416Jens Axboe2014-06-041-0/+2
| | | | | | | | | | | When the code was collapsed to avoid duplication, the recent patch for ensuring that a queue is idled before free was dropped, which was added by commit 19c5d84f14d2. Add back the blk_mq_tag_idle(), to ensure we don't leak a reference to an active queue when it is freed. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: handle NULL req return from blk_map_request in single queue modeJens Axboe2014-06-031-0/+2
| | | | | | | | blk_mq_map_request() can return NULL if we fail entering the queue (dying, or removed), in which case it has already ended IO on the bio. So nothing more to do, except just return. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: fix sparse warning on missed __percpu annotationMing Lei2014-06-031-1/+1
| | | | | | | | | | | | 'struct blk_mq_ctx' is __percpu, so add the annotation and fix the sparse warning reported from Fengguang: [block:for-linus 2/3] block/blk-mq.h:75:16: sparse: incorrect type in initializer (different address spaces) Reported-by: kbuild test robot <fengguang.wu@intel.com> Signed-off-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: fix schedule from atomic contextMing Lei2014-06-031-13/+23
| | | | | | | | | | | blk_mq_put_ctx() has to be called before io_schedule() in bt_get(). This patch fixes the problem by taking similar approach from percpu_ida allocation for the situation. Signed-off-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: move blk_mq_get_ctx/blk_mq_put_ctx to mq private headerMing Lei2014-06-031-22/+0
| | | | | | | The blk-mq tag code need these helpers. Signed-off-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: push IPI or local end_io decision to __blk_mq_complete_request()Jens Axboe2014-05-301-7/+13
| | | | | | | | | | | | | | We have callers outside of the blk-mq proper (like timeouts) that want to call __blk_mq_complete_request(), so rename the function and put the decision code for whether to use ->softirq_done_fn or blk_mq_endio() into __blk_mq_complete_request(). This also makes the interface more logical again. blk_mq_complete_request() attempts to atomically mark the request completed, and calls __blk_mq_complete_request() if successful. __blk_mq_complete_request() then just ends the request. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: remember to start timeout handler for direct queueJens Axboe2014-05-301-0/+1
| | | | | | | | Commit 07068d5b8e added a direct-to-hw-queue mode, but this mode needs to remember to add the request timeout handler as well. Without it, we don't track timeouts for these requests. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: make the sysfs mq/ layout reflect current mappingsJens Axboe2014-05-301-0/+4
| | | | | | | | | | | | | Currently blk-mq registers all the hardware queues in sysfs, regardless of whether it uses them (e.g. they have CPU mappings) or not. The unused hardware queues lack the cpux/ directories, and the other sysfs entries (like active, pending, etc) are all zeroes. Change this so that sysfs correctly reflects the current mappings of the hardware queues. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: blk_mq_tag_to_rq should handle flush requestShaohua Li2014-05-301-3/+9
| | | | | | | | | flush request is special, which borrows the tag from the parent request. Hence blk_mq_tag_to_rq needs special handling to return the flush request from the tag. Signed-off-by: Shaohua Li <shli@fusionio.com> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: request initialization optimizationsJens Axboe2014-05-291-17/+9
| | | | | | | | We currently clear a lot more than we need to, so make that a bit more clever. Make some of the init dependent on features, like only setting start_time if we are going to use it. Signed-off-by: Jens Axboe <axboe@fb.com>
* block: add queue flag for disabling SG mergingJens Axboe2014-05-291-0/+3
| | | | | | | | | | | | | If devices are not SG starved, we waste a lot of time potentially collapsing SG segments. Enough that 1.5% of the CPU time goes to this, at only 400K IOPS. Add a queue flag, QUEUE_FLAG_NO_SG_MERGE, which just returns the number of vectors in a bio instead of looping over all segments and checking for collapsible ones. Add a BLK_MQ_F_SG_MERGE flag so that drivers can opt-in on the sg merging, if they so desire. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: remove alloc_hctx and free_hctx methodsChristoph Hellwig2014-05-281-21/+5
| | | | | | | | There is no need for drivers to control hardware context allocation now that we do the context to node mapping in common code. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: add file comments and update copyright noticesJens Axboe2014-05-281-0/+6
| | | | | | | | None of the blk-mq files have an explanatory comment at the top for what that particular file does. Add that and add appropriate copyright notices as well. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: remove blk_mq_alloc_request_pinnedChristoph Hellwig2014-05-281-32/+16
| | | | | | | | We now only have one caller left and can open code it there in a cleaner way. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: do not use blk_mq_alloc_request_pinned in blk_mq_map_requestChristoph Hellwig2014-05-281-3/+5
| | | | | | | | We already do a non-blocking allocation in blk_mq_map_request, no need to repeat it. Just call __blk_mq_alloc_request to wait directly. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: remove blk_mq_wait_for_tagsChristoph Hellwig2014-05-281-7/+6
| | | | | | | | | | | | | The current logic for blocking tag allocation is rather confusing, as we first allocated and then free again a tag in blk_mq_wait_for_tags, just to attempt a non-blocking allocation and then repeat if someone else managed to grab the tag before us. Instead change blk_mq_alloc_request_pinned to simply do a blocking tag allocation itself and use the request we get back from it. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: initialize request in __blk_mq_alloc_requestChristoph Hellwig2014-05-281-32/+30
| | | | | | | | Both callers if __blk_mq_alloc_request want to initialize the request, so lift it into the common path. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: merge blk_mq_alloc_reserved_request into blk_mq_alloc_requestChristoph Hellwig2014-05-281-17/+3
| | | | | | | | Instead of having two almost identical copies of the same code just let the callers pass in the reserved flag directly. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: add helper to insert requests from irq contextChristoph Hellwig2014-05-281-1/+63
| | | | | | | | | | | | | | | Both the cache flush state machine and the SCSI midlayer want to submit requests from irq context, and the current per-request requeue_work unfortunately causes corruption due to sharing with the csd field for flushes. Replace them with a per-request_queue list of requests to be requeued. Based on an earlier test by Ming Lei. Signed-off-by: Christoph Hellwig <hch@lst.de> Reported-by: Ming Lei <tom.leiming@gmail.com> Tested-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: allow non-softirq completionsJens Axboe2014-05-271-3/+9
| | | | | | | | | | | | | | | | | | | Right now we export two ways of completing a request: 1) blk_mq_complete_request(). This uses an IPI (if needed) and completes through q->softirq_done_fn(). It also works with timeouts. 2) blk_mq_end_io(). This completes inline, and ignores any timeout state of the request. Let blk_mq_complete_request() handle non-softirq_done_fn completions as well, by just completing inline. If a driver has enough completion ports to place completions correctly, it need not define a mq_ops->complete() and we can avoid an indirect function call by doing the completion inline. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: pass in suggested NUMA node to ->alloc_hctx()Jens Axboe2014-05-271-11/+15
| | | | | | | | | | | Drivers currently have to figure this out on their own, and they are missing information to do it properly. The ones that did attempt to do it, do it wrong. So just pass in the suggested node directly to the alloc function. Signed-off-by: Jens Axboe <axboe@fb.com>
* block: only allocate/free mq_usage_counter in blk-mqMing Lei2014-05-271-0/+5
| | | | | | | | | The percpu counter is only used for blk-mq, so move its allocation and free inside blk-mq, and don't allocate it for legacy queue device. Signed-off-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: avoid code duplicationMing Lei2014-05-271-24/+37
| | | | | | | | blk_mq_exit_hw_queues() and blk_mq_free_hw_queues() are introduced to avoid code duplication. Signed-off-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: fix leak of hctx->ctx_mapMing Lei2014-05-271-0/+1
| | | | | | | hctx->ctx_map should have been freed inside blk_mq_free_queue(). Signed-off-by: Ming Lei <tom.leiming@gmail.com> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: idle all hardware contexts before freeing a queueChristoph Hellwig2014-05-261-0/+1
| | | | | | | | Without this we can leak the active_queues reference if a command is freed while it is considered active. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: allow setting of per-request timeoutsJens Axboe2014-05-231-2/+6
| | | | | | | | | Currently blk-mq uses the queue timeout for all requests. But for some commands, drivers may want to set a specific timeout for special requests. Allow this to be passed in through request->timeout, and use it if set. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: split make request handler for multi and single queueJens Axboe2014-05-221-50/+157
| | | | | | | | | | | | | | | | | We want slightly different behavior from them: - On single queue devices, we currently use the per-process plug for deferred IO and for merging. - On multi queue devices, we don't use the per-process plug, but we want to go straight to hardware for SYNC IO. Split blk_mq_make_request() into a blk_sq_make_request() for single queue devices, and retain blk_mq_make_request() for multi queue devices. Then we don't need multiple checks for q->nr_hw_queues in the request mapping. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: save memory by freeing requests on unused hardware queuesJens Axboe2014-05-211-52/+105
| | | | | | | | | | | | Depending on the topology of the machine and the number of queues exposed by a device, we can end up in a situation where some of the hardware queues are unused (as in, they don't map to any software queues). For this case, free up the memory used by the request map, as we will not use it. This can be a substantial amount of memory, depending on the number of queues vs CPUs and the queue depth of the device. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: allow the hctx cpu hotplug notifier to return errorsJens Axboe2014-05-211-4/+5
| | | | | | | Prepare this for the next patch which adds more smarts in the plugging logic, so that we can save some memory. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: Micro-optimize blk_queue_nomerges() checkRobert Elliott2014-05-201-1/+2
| | | | | | | | | In blk_mq_make_request(), do the blk_queue_nomerges() check outside the call to blk_attempt_plug_merge() to eliminate function call overhead when nomerges=2 (disabled) Signed-off-by: Robert Elliott <elliott@hp.com> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: initialize q->nr_requests after calling blk_queue_make_request()Jens Axboe2014-05-201-0/+5
| | | | | | | | blk_queue_make_requests() overwrites our set value for q->nr_requests, turning it into the default of 128. Set this appropriately after initializing queue values in blk_queue_make_request(). Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: allow changing of queue depth through sysfsJens Axboe2014-05-201-0/+22
| | | | | | | | | | | For request_fn based devices, the block layer exports a 'nr_requests' file through sysfs to allow adjusting of queue depth on the fly. Currently this returns -EINVAL for blk-mq, since it's not wired up. Wire this up for blk-mq, so that it now also always dynamic adjustments of the allowed queue depth for any given block device managed by blk-mq. Signed-off-by: Jens Axboe <axboe@fb.com>
* Merge branch 'for-3.16/blk-mq-tagging' into for-3.16/coreJens Axboe2014-05-191-6/+79
|\ | | | | | | | | | | | | Signed-off-by: Jens Axboe <axboe@fb.com> Conflicts: block/blk-mq-tag.c
| * blk-mq: improve support for shared tags mapsJens Axboe2014-05-131-6/+79
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds support for active queue tracking, meaning that the blk-mq tagging maintains a count of active users of a tag set. This allows us to maintain a notion of fairness between users, so that we can distribute the tag depth evenly without starving some users while allowing others to try unfair deep queues. If sharing of a tag set is detected, each hardware queue will track the depth of its own queue. And if this exceeds the total depth divided by the number of active queues, the user is actively throttled down. The active queue count is done lazily to avoid bouncing that data between submitter and completer. Each hardware queue gets marked active when it allocates its first tag, and gets marked inactive when 1) the last tag is cleared, and 2) the queue timeout grace period has passed. Signed-off-by: Jens Axboe <axboe@fb.com>
* | blk-mq: switch ctx pending map to the sparser blk_align_bitmapJens Axboe2014-05-191-28/+91
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Each hardware queue has a bitmap of software queues with pending requests. When new IO is queued on a software queue, the bit is set, and when IO is pruned on a hardware queue run, the bit is cleared. This causes a lot of traffic. Switch this from the regular BITS_PER_LONG bitmap to a sparser layout, similarly to what was done for blk-mq tagging. 20% performance increase was observed for single threaded IO, and about 15% performanc increase on multiple threads driving the same device. Signed-off-by: Jens Axboe <axboe@fb.com>
* | blk-mq: fix race in IO start accountingJens Axboe2014-05-091-4/+2
|/ | | | | | | | | Commit c6d600c6 opened up a small race where we could attempt to account IO completion on a request, racing with IO start accounting. Fix this up by ensuring that we've accounted for IO start before inserting the request. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: implement new and more efficient tagging schemeJens Axboe2014-05-091-12/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | blk-mq currently uses percpu_ida for tag allocation. But that only works well if the ratio between tag space and number of CPUs is sufficiently high. For most devices and systems, that is not the case. The end result if that we either only utilize the tag space partially, or we end up attempting to fully exhaust it and run into lots of lock contention with stealing between CPUs. This is not optimal. This new tagging scheme is a hybrid bitmap allocator. It uses two tricks to both be SMP friendly and allow full exhaustion of the space: 1) We cache the last allocated (or freed) tag on a per blk-mq software context basis. This allows us to limit the space we have to search. The key element here is not caching it in the shared tag structure, otherwise we end up dirtying more shared cache lines on each allocate/free operation. 2) The tag space is split into cache line sized groups, and each context will start off randomly in that space. Even up to full utilization of the space, this divides the tag users efficiently into cache line groups, avoiding dirtying the same one both between allocators and between allocator and freeer. This scheme shows drastically better behaviour, both on small tag spaces but on large ones as well. It has been tested extensively to show better performance for all the cases blk-mq cares about. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: initialize struct request fields individuallyChristoph Hellwig2014-05-091-2/+45
| | | | | | | | This allows us to avoid a non-atomic memset over ->atomic_flags as well as killing lots of duplicate initializations. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: update a hotplug comment for grammarJens Axboe2014-05-091-4/+4
| | | | Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: add basic round-robin of what CPU to queue workqueue work onJens Axboe2014-05-071-14/+31
| | | | | | | | Right now we just pick the first CPU in the mask, but that can easily overload that one. Add some basic batching and round-robin all the entries in the mask instead. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: remove extra requeue traceJens Axboe2014-05-021-4/+0
| | | | | | | | We already issue a blktrace requeue event in __blk_mq_requeue_request(), don't do it from the original caller as well. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: refactor request insertion/mergingJens Axboe2014-04-301-7/+15
| | | | | | | | | Refactor the logic around adding a new bio to a software queue, so we nest the ctx->lock where we really need it (merge and insertion) and don't hold it when we don't (init and IO start accounting). Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq remove debug BUG_ON() when draining software queuesJens Axboe2014-04-301-1/+0
| | | | | | It's never been of any use, lets get rid of it. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: fix waiting for reserved tagsJens Axboe2014-04-291-1/+1
| | | | | | | | | | | | blk_mq_wait_for_tags() is only able to wait for "normal" tags, not reserved tags. Pass in which one we should attempt to get a tag for, so that waiting for reserved tags will work. Reserved tags are used for internal commands, which are usually serialized. Hence no waiting generally takes place, but we should ensure that it actually works if users need that functionality. Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: respect rq_affinityChristoph Hellwig2014-04-251-2/+6
| | | | | | | | | | | | | | | | | | | | | | | The blk-mq code is using it's own version of the I/O completion affinity tunables, which causes a few issues: - the rq_affinity sysfs file doesn't work for blk-mq devices, even if it still is present, thus breaking existing tuning setups. - the rq_affinity = 1 mode, which is the defauly for legacy request based drivers isn't implemented at all. - blk-mq drivers don't implement any completion affinity with the default flag settings. This patches removes the blk-mq ipi_redirect flag and sysfs file, as well as the internal BLK_MQ_F_SHOULD_IPI flag and replaces it with code that respects the queue-wide rq_affinity flags and also implements the rq_affinity = 1 mode. This means I/O completion affinity can now only be tuned block-queue wide instead of per context, which seems more sensible to me anyway. Signed-off-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Jens Axboe <axboe@fb.com>
* blk-mq: fix race with timeouts and requeue eventsJens Axboe2014-04-241-7/+32
| | | | | | | | | | | | | If a requeue event races with a timeout, we can get into the situation where we attempt to complete a request from the timeout handler when it's not start anymore. This causes a crash. So have the timeout handler check that REQ_ATOM_STARTED is still set on the request - if not, we ignore the event. If this happens, the request has now been marked as complete. As a consequence, we need to ensure to clear REQ_ATOM_COMPLETE in blk_mq_start_request(), as to maintain proper request state. Signed-off-by: Jens Axboe <axboe@fb.com>
* Revert "blk-mq: initialize req->q in allocation"Jens Axboe2014-04-241-3/+1
| | | | | | | | This reverts commit 6a3c8a3ac0e68dcfc2a01f4aa1ca0edd1a1701eb. We need selective clearing of the request to make the init-at-free time completely safe. Otherwise we end up stomping on rq->atomic_flags, which we don't want to do.
OpenPOWER on IntegriCloud