.\" Copyright (c) 2010-2013 Advanced Computing Technologies LLC .\" Written by: John H. Baldwin .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ .\" .Dd October 29, 2014 .Dt ETCUPDATE 8 .Os .Sh NAME .Nm etcupdate .Nd "manage updates to system files not updated by installworld" .Sh SYNOPSIS .Nm .Op Fl npBF .Op Fl d Ar workdir .Op Fl r | Fl s Ar source | Fl t Ar tarball .Op Fl A Ar patterns .Op Fl D Ar destdir .Op Fl I Ar patterns .Op Fl L Ar logfile .Op Fl M Ar options .Nm .Cm build .Op Fl B .Op Fl d Ar workdir .Op Fl s Ar source .Op Fl L Ar logfile .Op Fl M Ar options .Ar tarball .Nm .Cm diff .Op Fl d Ar workdir .Op Fl D Ar destdir .Op Fl I Ar patterns .Op Fl L Ar logfile .Nm .Cm extract .Op Fl B .Op Fl d Ar workdir .Op Fl s Ar source | Fl t Ar tarball .Op Fl L Ar logfile .Op Fl M Ar options .Nm .Cm resolve .Op Fl p .Op Fl d Ar workdir .Op Fl D Ar destdir .Op Fl L Ar logfile .Nm .Cm status .Op Fl d Ar workdir .Op Fl D Ar destdir .Sh DESCRIPTION The .Nm utility is a tool for managing updates to files that are not updated as part of .Sq make installworld such as files in .Pa /etc . It manages updates by doing a three-way merge of changes made to these files against the local versions. It is also designed to minimize the amount of user intervention with the goal of simplifying upgrades for clusters of machines. .Pp To perform a three-way merge, .Nm keeps copies of the current and previous versions of files that it manages. These copies are stored in two trees known as the .Dq current and .Dq previous trees. During a merge, .Nm compares the .Dq current and .Dq previous copies of each file to determine which changes need to be merged into the local version of each file. If a file can be updated without generating a conflict, .Nm will update the file automatically. If the local changes to a file conflict with the changes made to a file in the source tree, then a merge conflict is generated. The conflict must be resolved after the merge has finished. The .Nm utility will not perform a new merge until all conflicts from an earlier merge are resolved. .Sh MODES The .Nm utility supports several modes of operation. The mode is specified via an optional command argument. If present, the command must be the first argument on the command line. If a command is not specified, the default mode is used. .Ss Default Mode The default mode merges changes from the source tree to the destination directory. First, it updates the .Dq current and .Dq previous trees. Next, it compares the two trees merging changes into the destination directory. Finally, it displays warnings for any conditions it could not handle automatically. .Pp If the .Fl r option is not specified, then the first step taken is to update the .Dq current and .Dq previous trees. If a .Dq current tree already exists, then that tree is saved as the .Dq previous tree. An older .Dq previous tree is removed if it exists. By default the new .Dq current tree is built from a source tree. However, if a tarball is specified via the .Fl t option, then the tree is extracted from that tarball instead. .Pp Next, .Nm compares the files in the .Dq current and .Dq previous trees. If a file was removed from the .Dq current tree, then it will be removed from the destination directory only if it does not have any local modifications. If a file was added to the .Dq current tree, then it will be copied to the destination directory only if it would not clobber an existing file. If a file is changed in the .Dq current tree, then .Nm will attempt to merge the changes into the version of the file in the destination directory. If the merge encounters conflicts, then a version of the file with conflict markers will be saved for future resolution. If the merge does not encounter conflicts, then the merged version of the file will be saved in the destination directory. If .Nm is not able to safely merge in changes to a file other than a merge conflict, it will generate a warning. .Pp For each file that is updated a line will be output with a leading character to indicate the action taken. The possible actions follow: .Pp .Bl -tag -width "A" -compact -offset indent .It A Added .It C Conflict .It D Deleted .It M Merged .It U Updated .El .Pp Finally, if any warnings were encountered they are displayed after the merge has completed. .Pp Note that for certain files .Nm will perform post-install actions any time that the file is updated. Specifically, .Xr pwd_mkdb 8 is invoked if .Pa /etc/master.passwd is changed, .Xr cap_mkdb 1 is invoked to update .Pa /etc/login.conf.db if .Pa /etc/login.conf is changed, .Xr newaliases 1 is invoked if .Pa /etc/mail/aliases is changed, and .Pa /etc/rc.d/motd is invoked if .Pa /etc/motd is changed. One exception is that if .Pa /etc/mail/aliases is changed and the destination directory is not the default, then a warning will be issued instead. This is due to a limitation of the .Xr newaliases 1 command. Similarly, if .Pa /etc/motd is changed and the destination directory is not the default, then .Pa /etc/rc.d/motd will not be executed due to a limitation of that script. In this case no warning is issued as the result of .Pa /etc/rc.d/motd is merely cosmetic and will be corrected on the next reboot. .Ss Build Mode The .Cm build mode is used to build a tarball that contains a snapshot of a .Dq current tree. This tarball can be used by the default and extract modes. Using a tarball can allow .Nm to perform a merge without requiring a source tree that matches the currently installed world. The .Fa tarball argument specifies the name of the file to create. The file will be a .Xr tar 5 file compressed with .Xr bzip2 1 . .Ss Diff Mode The .Cm diff mode compares the versions of files in the destination directory to the .Dq current tree and generates a unified format diff of the changes. This can be used to determine which files have been locally modified and how. Note that .Nm does not manage files that are not maintained in the source tree such as .Pa /etc/fstab and .Pa /etc/rc.conf . .Ss Extract Mode The .Cm extract mode generates a new .Dq current tree. Unlike the default mode, it does not save any existing .Dq current tree and does not modify any existing .Dq previous tree. The new .Dq current tree can either be built from a source tree or extracted from a tarball. .Ss Resolve Mode The .Cm resolve mode is used to resolve any conflicts encountered during a merge. In this mode, .Nm iterates over any existing conflicts prompting the user for actions to take on each conflicted file. For each file, the following actions are available: .Pp .Bl -tag -width "(tf) theirs-full" -compact .It (p) postpone Ignore this conflict for now. .It (df) diff-full Show all changes made to the merged file as a unified diff. .It (e) edit Change the merged file in an editor. .It (r) resolved Install the merged version of the file into the destination directory. .It (mf) mine-full Use the version of the file in the destination directory and ignore any changes made to the file in the .Dq current tree. .It (tf) theirs-full Use the version of the file from the .Dq current tree and discard any local changes made to the file. .It (h) help Display the list of commands. .El .Ss Status Mode The .Cm status mode shows a summary of the results of the most recent merge. First it lists any files for which there are unresolved conflicts. Next it lists any warnings generated during the last merge. If the last merge did not generate any conflicts or warnings, then nothing will be output. .Sh OPTIONS The following options are available. Note that most options do not apply to all modes. .Bl -tag -width ".Fl A Ar patterns" .It Fl A Ar patterns Always install the new version of any files that match any of the patterns listed in .Ar patterns . Each pattern is evaluated as an .Xr sh 1 shell pattern. This option may be specified multiple times to specify multiple patterns. Multiple space-separated patterns may also be specified in a single option. Note that ignored files specified via the .Ev IGNORE_FILES variable or the .Fl I option will not be installed. .It Fl B Do not build generated files in a private object tree. Instead, reuse the generated files from a previously built object tree that matches the source tree. This can be useful to avoid gratuitous conflicts in .Xr sendmail 8 configuration files when bootstrapping. It can also be useful for building a tarball that matches a specific world build. .It Fl D Ar destdir Specify an alternate destination directory as the target of a merge. This is analogous to the .Dv DESTDIR variable used with .Sq make installworld . The default destination directory is an empty string which results in merges updating .Pa /etc on the local machine. .It Fl d Ar workdir Specify an alternate directory to use as the work directory. The work directory is used to store the .Dq current and .Dq previous trees as well as unresolved conflicts. The default work directory is .Pa /var/db/etcupdate . .It Fl F Ignore changes in the FreeBSD ID string when comparing files in the destination directory to files in either of the .Dq current or .Dq previous trees. In .Cm diff mode, this reduces noise due to FreeBSD ID string changes in the output. During an update this can simplify handling for harmless conflicts caused by FreeBSD ID string changes. .Pp Specifically, if a file in the destination directory is identical to the same file in the .Dq previous tree modulo the FreeBSD ID string, then the file is treated as if it was unmodified and the .Dq current version of the file will be installed. Similarly, if a file in the destination directory is identical to the same file in the .Dq current tree modulo the FreeBSD ID string, then the .Dq current version of the file will be installed to update the ID string. If the .Dq previous and .Dq current versions of the file are identical, then .Nm will not change the file in the destination directory. .Pp Due to limitations in the .Xr diff 1 command, this option may not have an effect if there are other changes in a file that are close to the FreeBSD ID string. .It Fl I Ar patterns Ignore any files that match any of the patterns listed in .Ar patterns . No warnings or other messages will be generated for those files during a merge. Each pattern is evaluated as an .Xr sh 1 shell pattern. This option may be specified multiple times to specify multiple patterns. Multiple space-separated patterns may also be specified in a single option. .It Fl L Ar logfile Specify an alternate path for the log file. The .Nm utility logs each command that it invokes along with the standard output and standard error to this file. By default the log file is stored in a file named .Pa log in the work directory. .It Fl M Ar options Pass .Ar options as additional parameters to .Xr make 1 when building a .Dq current tree. This can be used for to set the .Dv TARGET or .Dv TARGET_ARCH variables for a cross-build. .It Fl n Enable .Dq dry-run mode. Do not merge any changes to the destination directory. Instead, report what actions would be taken during a merge. Note that the existing .Dq current and .Dq previous trees will not be changed. If the .Fl r option is not specified, then a temporary .Dq current tree will be extracted to perform the comparison. .It Fl p Enable .Dq pre-world mode. Only merge changes to files that are necessary to successfully run .Sq make installworld or .Sq make installkernel . When this flag is enabled, the existing .Dq current and .Dq previous trees are left alone. Instead, a temporary tree is populated with the necessary files. This temporary tree is compared against the .Dq current tree. This allows a normal update to be run after .Sq make installworld has completed. Any conflicts generated during a .Dq pre-world update should be resolved by a .Dq pre-world .Cm resolve . .It Fl r Do not update the .Dq current and .Dq previous trees during a merge. This can be used to .Dq re-run a previous merge operation. .It Fl s Ar source Specify an alternate source tree to use when building or extracting a .Dq current tree. The default source tree is .Pa /usr/src . .It Fl t Ar tarball Extract a new .Dq current tree from a tarball previously generated by the .Cm build command rather than building the tree from a source tree. .El .Sh CONFIG FILE The .Nm utility can also be configured by setting variables in an optional configuration file named .Pa /etc/etcupdate.conf . Note that command line options override settings in the configuration file. The configuration file is executed by .Xr sh 1 , so it uses that syntax to set configuration variables. The following variables can be set: .Bl -tag -width ".Ev ALWAYS_INSTALL" .It Ev ALWAYS_INSTALL Always install files that match any of the patterns listed in this variable similar to the .Fl A option. .It Ev DESTDIR Specify an alternate destination directory similar to the .Fl D option. .It Ev EDITOR Specify a program to edit merge conflicts. .It Ev FREEBSD_ID Ignore changes in the FreeBSD ID string similar to the .Fl F option. This is enabled by setting the variable to a non-empty value. .It Ev IGNORE_FILES Ignore files that match any of the patterns listed in this variable similar to the .Fl I option. .It Ev LOGFILE Specify an alternate path for the log file similar to the .Fl L option. .It Ev MAKE_OPTIONS Pass additional options to .Xr make 1 when building a .Dq current tree similar to the .Fl M option. .It Ev SRCDIR Specify an alternate source tree similar to the .Fl s option. .It Ev WORKDIR Specify an alternate work directory similar to the .Fl d option. .El .Sh ENVIRONMENT The .Nm utility uses the program identified in the .Ev EDITOR environment variable to edit merge conflicts. If .Ev EDITOR is not set, .Xr vi 1 is used as the default editor. .Sh FILES .Bl -tag -width ".Pa /var/db/etcupdate/log" -compact .It Pa /etc/etcupdate.conf Optional config file. .It Pa /var/db/etcupdate Default work directory used to store trees and other data. .It Pa /var/db/etcupdate/log Default log file. .El .Sh EXIT STATUS .Ex -std .Sh EXAMPLES To compare the files in .Pa /etc with the stock versions: .Pp .Dl "etcupdate diff" .Pp To merge changes after an upgrade via the buildworld and installworld process: .Pp .Dl "etcupdate" .Pp To resolve any conflicts generated during a merge: .Pp .Dl "etcupdate resolve" .Ss Bootstrapping The .Nm utility may need to be bootstrapped before it can be used. The .Cm diff command will fail with an error about a missing reference tree if bootstrapping is needed. .Pp Bootstrapping .Nm requires a source tree that matches the currently installed world. The easiest way to ensure this is to bootstrap .Nm before updating the source tree to start the next world upgrade cycle. First, generate a reference tree: .Pp .Dl "etcupdate extract" .Pp Second, use the .Cm diff command to compare the reference tree to your current files in .Pa /etc . Undesired differences should be removed using an editor, .Xr patch 1 , or by copying files from the reference tree .Po located at .Pa /var/db/etcupdate/current by default .Pc . .Pp If the tree at .Pa /usr/src is already newer than the currently installed world, a new tree matching the currently installed world can be checked out to a temporary location. The reference tree for .Nm can then be generated via: .Pp .Dl "etcupdate extract -s /path/to/tree" .Pp The .Cm diff command can be used as above to remove undesired differences. Afterwards, the changes in the tree at .Pa /usr/src can be merged via a regular merge. .Sh DIAGNOSTICS The following warning messages may be generated during a merge. Note that several of these warnings cover obscure cases that should occur rarely if at all in practice. For example, if a file changes from a file to a directory in the .Dq current tree and the file was modified in the destination directory, then a warning will be triggered. In general, when a warning references a pathname, the corresponding file in the destination directory is not changed by a merge operation. .Bl -diag .It "Directory mismatch: ()" An attempt was made to create a directory at .Pa path but an existing file of type .Dq type already exists for that path name. .It "Modified link changed: ( became )" The target of a symbolic link named .Pa file was changed from .Dq old to .Dq new in the .Dq current tree. The symbolic link has been modified to point to a target that is neither .Dq old nor .Dq new in the destination directory. .It "Modified mismatch: ( vs )" A file named .Pa file of type .Dq new was modified in the .Dq current tree, but the file exists as a different type .Dq dest in the destination directory. .It "Modified changed: ( became )" A file named .Pa file changed type from .Dq old in the .Dq previous tree to type .Dq new in the .Dq current tree. The file in the destination directory of type .Dq type has been modified, so it could not be merged automatically. .It "Modified remains: " The file of type .Dq type named .Pa file has been removed from the .Dq current tree, but it has been locally modified. The modified version of the file remains in the destination directory. .It "Needs update: /etc/localtime (required manual update via tzsetup(1))" The .Fa /var/db/zoneinfo file does not exist, so .Nm was not able to refresh .Fa /etc/localtime from its source file in .Fa /usr/share/zoneinfo . Running .Xr tzsetup 1 will both refresh .Fa /etc/localtime and generate .Fa /var/db/zoneinfo permitting future updates to refresh .Fa /etc/localtime automatically. .It "Needs update: /etc/mail/aliases.db (required manual update via newaliases(1))" The file .Pa /etc/mail/aliases was updated during a merge with a non-empty destination directory. Due to a limitation of the .Xr newaliases 1 command, .Nm was not able to automatically update the corresponding aliases database. .It "New file mismatch: ( vs )" A new file named .Pa file of type .Dq new has been added to the .Dq current tree. A file of that name already exists in the destination directory, but it is of a different type .Dq dest . .It "New link conflict: ( vs )" A symbolic link named .Pa file has been added to the .Dq current tree that links to .Dq new . A symbolic link of the same name already exists in the destination directory, but it links to a different target .Dq dest . .It "Non-empty directory remains: " The directory .Pa file was removed from the .Dq current tree, but it contains additional files in the destination directory. These additional files as well as the directory remain. .It "Remove mismatch: ( became )" A file named .Pa file changed from type .Dq old in the .Dq previous tree to type .Dq new in the .Dq current tree, but it has been removed in the destination directory. .It "Removed file changed: " A file named .Pa file was modified in the .Dq current tree, but it has been removed in the destination directory. .It "Removed link changed: ( became )" The target of a symbolic link named .Pa file was changed from .Dq old to .Dq new in the .Dq current tree, but it has been removed in the destination directory. .El .Sh SEE ALSO .Xr cap_mkdb 1 , .Xr diff 1 , .Xr make 1 , .Xr newaliases 1 , .Xr sh 1 , .Xr pwd_mkdb 8 .Sh HISTORY The .Nm utility first appeared in .Fx 10.0 . .Sh AUTHORS The .Nm utility was written by .An John Baldwin Aq jhb@FreeBSD.org . .Sh BUGS Rerunning a merge does not automatically delete conflicts left over from a previous merge. Any conflicts must be resolved before the merge can be rerun. It it is not clear if this is a feature or a bug. .Pp There is no way to easily automate conflict resolution for specific files. For example, one can imagine a syntax along the lines of .Pp .Dl "etcupdate resolve tf /some/file" .Pp to resolve a specific conflict in an automated fashion. .Pp It might be nice to have something like a .Sq revert command to replace a locally modified version of a file with the stock version of the file. For example: .Pp .Dl "etcupdate revert /etc/mail/freebsd.cf" .Pp Bootstrapping .Nm often results in gratuitous diffs in .Pa /etc/mail/*.cf that cause conflicts in the first merge. If an object tree that matches the source tree is present when bootstrapping, then passing the .Fl B flag to the .Cm extract command can work around this.