summaryrefslogtreecommitdiffstats
path: root/sbin/hastd/primary.c
Commit message (Collapse)AuthorAgeFilesLines
* MFC r257155, r257582, r259191, r259192, r259193, r259194, r259195, r259196:trociny2013-12-281-134/+109
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | r257155: Make hastctl list command output current queue sizes. Reviewed by: pjd r257582 (pjd): Correct alignment. r259191: For memsync replication, hio_countdown is used not only as an indication when a request can be moved to done queue, but also for detecting the current state of memsync request. This approach has problems, e.g. leaking a request if memsynk ack from the secondary failed, or racy usage of write_complete, which should be called only once per write request, but for memsync can be entered by local_send_thread and ggate_send_thread simultaneously. So the following approach is implemented instead: 1) Use hio_countdown only for counting components we waiting to complete, i.e. initially it is always 2 for any replication mode. 2) To distinguish between "memsync ack" and "memsync fin" responses from the secondary, add and use hio_memsyncacked field. 3) write_complete() in component threads is called only before releasing hio_countdown (i.e. before the hio may be returned to the done queue). 4) Add and use hio_writecount refcounter to detect when write_complete() can be called in memsync case. Reported by: Pete French petefrench ingresso.co.uk Tested by: Pete French petefrench ingresso.co.uk r259192: Add some macros to make the code more readable (no functional chages). r259193: Fix compiler warnings. r259194: In remote_send_thread, if sending a request fails don't take the request back from the receive queue -- it might already be processed by remote_recv_thread, which lead to crashes like below: (primary) Unable to receive reply header: Connection reset by peer. (primary) Unable to send request (Connection reset by peer): WRITE(954662912, 131072). (primary) Disconnected from kopusha:7772. (primary) Increasing localcnt to 1. (primary) Assertion failed: (old > 0), function refcnt_release, file refcnt.h, line 62. Taking the request back was not necessary (it would properly be processed by the remote_recv_thread) and only complicated things. r259195: Send wakeup to threads waiting on empty queue before releasing the lock to decrease spurious wakeups. Submitted by: davidxu r259196: Check remote protocol version only for the first connection (when it is actually sent by the remote node). Otherwise it generated confusing "Negotiated protocol version 1" debug messages when processing the second connection.
* MFC r257154:trociny2013-10-311-1/+1
| | | | | | | | | | | | Merging local and remote bitmaps must be protected by hr_amp lock. This is believed to fix hastd crashes, which might occur during synchronization, triggered by the failed assertion: Assertion failed: (amp->am_memtab[ext] > 0), function activemap_write_complete, file activemap.c, line 351. Approved by: re (glebius)
* When updating the map of dirty extents, most recently used extents aretrociny2013-09-191-8/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | kept dirty to reduce the number of on-disk metadata updates. The sequence of operations is: 1) acquire the activemap lock; 2) update in-memory map; 3) if the list of keepdirty extents is changed, update on-disk metadata; 4) release the lock. On-disk updates are not frequent in comparison with in-memory updates, while require much more time. So situations are possible when one thread is updating on-disk metadata and another one is waiting for the activemap lock just to update the in-memory map. Improve this by introducing additional, on-disk map lock: when in-memory map is updated and it is detected that the on-disk map needs update too, the on-disk map lock is acquired and the on-memory lock is released before flushing the map. Reported by: Yamagi Burmeister yamagi.org Tested by: Yamagi Burmeister yamagi.org Reviewed by: pjd Approved by: re (marius) MFC after: 2 weeks
* Use cv_broadcast() instead of cv_signal() when waking up threadstrociny2013-09-191-2/+2
| | | | | | | | | | | | | | | | | | | | waiting on an empty queue as the queue may have several consumers. Before the fix the following scenario was possible: 2 threads are waiting on empty queue, 2 threads are inserting simultaneously. The first inserting thread detects that the queue is empty and is going to send the signal, but before it sends the second thread inserts too. When the first sends the signal only one of the waiting threads receive it while the other one may wait forever. The scenario above is is believed to be the cause of the observed cases, when ggate_recv_thread() was getting stuck on taking free request, while the free queue was not empty. Reviewed by: pjd Tested by: Yamagi Burmeister yamagi.org Approved by: re (marius) MFC after: 2 weeks
* Use C11 <stdatomic.h> instead of our non-standard <machine/atomic.h>.ed2013-04-271-6/+7
| | | | Reviewed by: pjd
* Add i/o error counters to hastd(8) and make hastctl(8) displaytrociny2013-02-251-0/+18
| | | | | | | them. This may be useful for detecting problems with HAST disks. Discussed with and reviewed by: pjd MFC after: 1 week
* - Add support for 'memsync' mode. This is the fastest replication mode that'spjd2013-02-171-21/+165
| | | | | | | | | | why it will now be the default. - Bump protocol version to 2 and add backward compatibility for version 1. - Allow to specify hosts by kern.hostid as well (in addition to hostname and kern.hostuuid) in configuration file. Sponsored by: Panzura Tested by: trociny
* Make use of GEOM Gate direct reads feature. This allows HAST to servepjd2012-07-041-1/+31
| | | | | | | | | | | | | | reads with native speed of the underlying provider. There are three situations when direct reads are not used: 1. Data is being synchronized and synchronization source is the secondary node, which means secondary node has more recent data and we should read from it. 2. Local read failed and we have to try to read from the secondary node. 3. Local component is unavailable and all I/O requests are served from the secondary node. Sponsored by: Panzura, http://www.panzura.com MFC after: 1 month
* Simplify the code by using snprlcat().pjd2012-06-031-27/+24
| | | | MFC after: 3 days
* If a local write request is from the synchronization thread, when ittrociny2012-02-051-1/+2
| | | | | | | | | | | | | is synchronizing data that is out of date on the local component, we should not send G_GATE_CMD_DONE acknowledge to the kernel. This fixes the issue, observed in async mode, when on synchronization from the remote component the worker terminated with "G_GATE_CMD_DONE failed" error. Reported by: Artem Kajalainen <artem kayalaynen ru> Reviewed by: pjd MFC after: 1 week
* Fix the regression introduced in r226859: if the local component istrociny2012-02-051-1/+1
| | | | | | | | out of date BIO_READ requests got lost instead of being sent to the remote component. Reviewed by: pjd MFC after: 1 week
* For functions that return -1 on failure check exactly for -1 and not forpjd2012-01-101-38/+38
| | | | | | any negative number. MFC after: 3 days
* Spelling fixes for sbin/uqs2012-01-071-1/+1
|
* fork(2) returns -1 on failure, not some random negative number.pjd2012-01-061-1/+1
| | | | MFC after: 3 days
* Implement 'async' mode for HAST.pjd2011-10-271-47/+78
| | | | MFC after: 3 days
* Minor cleanups.pjd2011-10-271-1/+2
| | | | MFC after: 3 days
* Reduce indentation.pjd2011-10-271-24/+24
| | | | MFC after: 3 days
* Improve comment so it doesn't suggest race is possible, but that we handlepjd2011-10-271-1/+1
| | | | | | the race. MFC after: 3 days
* Monor cleanups.pjd2011-10-271-6/+7
| | | | MFC after: 3 days
* Delay resuid generation until first connection to secondary, not until firstpjd2011-10-271-6/+3
| | | | | | | write. This way on first connection we will synchronize only the extents that were modified during the lifetime of primary node, not entire GEOM provider. MFC after: 3 days
* Correct typo.pjd2011-09-281-1/+1
| | | | MFC after: 3 days
* If the underlying provider doesn't support BIO_FLUSH, log it only oncepjd2011-09-281-0/+7
| | | | | | and don't bother trying in the future. MFC after: 3 days
* After every activemap change flush disk's write cache, so that writepjd2011-09-281-0/+19
| | | | | | | | | | | | reordering won't make the actual write to be committed before marking the coresponding extent as dirty. It can be disabled in configuration file. If BIO_FLUSH is not supported by the underlying file system we log a warning and never send BIO_FLUSH again to that GEOM provider. MFC after: 3 days
* No need to wrap pjdlog functions around with KEEP_ERRNO() macro.pjd2011-09-271-2/+1
| | | | MFC after: 3 days
* Correct two mistakes when converting asserts to PJDLOG_ASSERT()/PJDLOG_ABORT().pjd2011-09-271-4/+2
| | | | MFC after: 3 days
* Fix indentation.trociny2011-07-131-1/+1
| | | | Approved by: pjd (mentor)
* Check the returned value of activemap_write_complete() and update matadata ontrociny2011-06-281-2/+5
| | | | | | | | disk if needed. This should fix a potential case when extents are cleared in activemap but metadata is not updated on disk. Suggested by: pjd Approved by: pjd (mentor)
* In HAST we use two sockets - one for only sending the data and one fortrociny2011-06-171-0/+2
| | | | | | | | | | | | | | | | | | only receiving the data. In r220271 the unused directions were disabled using shutdown(2). Unfortunately, this broke automatic receive buffer sizing, which currently works only for connections in ETASBLISHED state. It was a root cause of the issue reported by users, when connection between primary and secondary could get stuck. Disable the code introduced in r220271 until the issue with automatic buffer sizing is not resolved. Reported by: Daniel Kalchev <daniel@digsys.bg>, danger, sobomax Tested by: Daniel Kalchev <daniel@digsys.bg>, danger Approved by: pjd (mentor) MFC after: 1 week
* If READ from the local node failed we send the request to the remotetrociny2011-05-291-1/+1
| | | | | | | node. There is no use in doing this for synchronization requests. Approved by: pjd (mentor) MFC after: 1 week
* Keep statistics on number of BIO_READ, BIO_WRITE, BIO_DELETE and BIO_FLUSHpjd2011-05-231-0/+11
| | | | | | | | | | | requests as well as number of activemap updates. Number of BIO_WRITEs and activemap updates are especially interesting, because if those two are too close to each other, it means that your workload needs bigger number of dirty extents. Activemap should be updated as rarely as possible. MFC after: 1 week
* Currently we are unable to use capsicum for the primary worker process,pjd2011-05-141-1/+1
| | | | | | | | | | | | | | | | | because we need to do ioctl(2)s, which are not permitted in the capability mode. What we do now is to chroot(2) to /var/empty, which restricts access to file system name space and we drop privileges to hast user and hast group. This still allows to access to other name spaces, like list of processes, network and sysvipc. To address that, use jail(2) instead of chroot(2). Using jail(2) will restrict access to process table, network (we use ip-less jails) and sysvipc (if security.jail.sysvipc_allowed is turned off). This provides much better separation. MFC after: 1 week
* When we become primary, we connect to the remote and expect it to be inpjd2011-04-201-13/+49
| | | | | | | | | | secondary role. It is possible that the remote node is primary, but only because there was a role change and it didn't finish cleaning up (unmounting file systems, etc.). If we detect such situation, wait for the remote node to switch the role to secondary before accepting I/Os. If we don't wait for it in that case, we will most likely cause split-brain. MFC after: 1 week
* Scenario:pjd2011-04-191-0/+19
| | | | | | | | | | | | | | | | | | | - We have two nodes connected and synchronized (local counters on both sides are 0). - We take secondary down and recreate it. - Primary connects to it and starts synchronization (but local counters are still 0). - We switch the roles. - Synchronization restarts but data is synchronized now from new primary (because local counters are 0) that doesn't have new data yet. This fix this issue we bump local counter on primary when we discover that connected secondary was recreated and has no data yet. Reported by: trociny Discussed with: trociny Tested by: trociny MFC after: 1 week
* Declare directions for sockets between primary and secondary.pjd2011-04-021-0/+5
| | | | | | | In HAST we use two sockets - one for only sending the data and one for only receiving the data. MFC after: 1 month
* Handle the problem described in r220264 by using GEOM GATE queue of unlimitedpjd2011-04-021-1/+1
| | | | | | length. This should fix deadlocks reported by HAST users. MFC after: 1 week
* Use timeout from configuration file not only when sending and receiving,pjd2011-03-251-1/+1
| | | | | | but also when establishing connection. MFC after: 1 week
* Use role2str() when setting process title.pjd2011-03-251-1/+1
| | | | MFC after: 1 week
* After synchronization is complete we should make primary counters betrociny2011-03-221-3/+3
| | | | | | | | | | | | | equal to secondary counters: primary_localcnt = secondary_remotecnt primary_remotecnt = secondary_localcnt Previously it was done wrong and split-brain was observed after primary had synchronized up-to-date data from secondary. Approved by: pjd (mentor) MFC after: 1 week
* For requests that are sent only to remote component use thetrociny2011-03-221-2/+7
| | | | | | error from remote. Approved by: pjd (mentor) MFC after: 1 week
* White space cleanups.pjd2011-03-221-3/+3
| | | | MFC after: 1 week
* When dropping privileges prefer capsicum over chroot+setgid+setuid.pjd2011-03-211-1/+1
| | | | | | | | | We can use capsicum for secondary worker processes and hastctl. When working as primary we drop privileges using chroot+setgid+setuid still as we need to send ioctl(2)s to ggate device, for which capsicum doesn't allow (yet). X-MFC after: capsicum is merged to stable/8
* Initialize localcnt on first write. This fixes assertion when we createpjd2011-03-211-2/+6
| | | | | | | resource, set role to primary, do no writes, then sent it to secondary and accept connection from primary. MFC after: 1 week
* In hast.conf we define the other node's address in 'remote' variable.pjd2011-03-211-11/+21
| | | | | | | | | | | | | | | | | | This way we know how to connect to secondary node when we are primary. The same variable is used by the secondary node - it only accepts connections from the address stored in 'remote' variable. In cluster configurations it is common that each node has its individual IP address and there is one addtional shared IP address which is assigned to primary node. It seems it is possible that if the shared IP address is from the same network as the individual IP address it might be choosen by the kernel as a source address for connection with the secondary node. Such connection will be rejected by secondary, as it doesn't come from primary node individual IP. Add 'source' variable that allows to specify source IP address we want to bind to before connecting to the secondary node. MFC after: 1 week
* For secondary, set 2 * HAST_KEEPALIVE seconds timeout for incomingtrociny2011-03-171-8/+4
| | | | | | | | | | connection so the worker will exit if it does not receive packets from the primary during this interval. Reported by: Christian Vogt <Christian.Vogt@haw-hamburg.de> Tested by: Christian Vogt <Christian.Vogt@haw-hamburg.de> Approved by: pjd (mentor) MFC after: 1 week
* Make workers inherit debug level from the main process.trociny2011-03-111-1/+3
| | | | | Approved by: pjd (mentor) MFC after: 1 week
* - Log size of data to synchronize in human readable form (using %N).pjd2011-03-071-6/+18
| | | | | | | - Log synchronization time (using %T). - Log synchronization speed in human readable form (using %N). MFC after: 2 weeks
* Allow to compress on-the-wire data using two algorithms:pjd2011-03-061-0/+8
| | | | | | | | | | | - HOLE - it simply turns all-zero blocks into few bytes header; it is extremely fast, so it is turned on by default; it is mostly intended to speed up initial synchronization where we expect many zeros; - LZF - very fast algorithm by Marc Alexander Lehmann, which shows very decent compression ratio and has BSD license. MFC after: 2 weeks
* Allow to checksum on-the-wire data using either CRC32 or SHA256.pjd2011-03-061-10/+19
| | | | MFC after: 2 weeks
* Setup another socketpair between parent and child, so that primary sandboxedpjd2011-02-031-30/+60
| | | | | | | | | worker can ask the main privileged process to connect in worker's behalf and then we can migrate descriptor using this socketpair to worker. This is not really needed now, but will be needed once we start to use capsicum for sandboxing. MFC after: 1 week
* Add missing locking after moving keepalive_send() to remote send threadpjd2011-02-031-3/+9
| | | | | | in r214692. MFC after: 1 week
OpenPOWER on IntegriCloud