$FreeBSD$ Design of xlocale ================= The xlocale APIs come from Darwin, although a subset is now part of POSIX 2008. They fall into two broad categories: - Manipulation of per-thread locales (POSIX) - Locale-aware functions taking an explicit locale argument (Darwin) This document describes the implementation of these APIs for FreeBSD. Goals ----- The overall goal of this implementation is to be compatible with the Darwin version. Additionally, it should include minimal changes to the existing locale code. A lot of the existing locale code originates with 4BSD or earlier and has had over a decade of testing. Replacing this code, unless absolutely necessary, gives us the potential for more bugs without much benefit. With this in mind, various libc-private functions have been modified to take a locale_t parameter. This causes a compiler error if they are accidentally called without a locale. This approach was taken, rather than adding _l variants of these functions, to make it harder for accidental uses of the global-locale versions to slip in. Locale Objects -------------- A locale is encapsulated in a `locale_t`, which is an opaque type: a pointer to a `struct _xlocale`. The name `_xlocale` is unfortunate, as it does not fit well with existing conventions, but is used because this is the name the Darwin implementation gives to this structure and so may be used by existing (bad) code. This structure should include all of the information corresponding to a locale. A locale_t is almost immutable after creation. There are no functions that modify it, and it can therefore be used without locking. It is the responsibility of the caller to ensure that a locale is not deallocated during a call that uses it. Each locale contains a number of components, one for each of the categories supported by `setlocale()`. These are likewise immutable after creation. This differs from the Darwin implementation, which includes a deprecated `setinvalidrune()` function that can modify the rune locale. The exception to these mutability rules is a set of `mbstate_t` flags stored with each locale. These are used by various functions that previously had a static local `mbstate_t` variable. The components are reference counted, and so can be aliased between locale objects. This makes copying locales very cheap. The Global Locale ----------------- All locales and locale components are reference counted. The global locale, however, is special. It, and all of its components, are static and so no malloc() memory is required when using a single locale. This means that threads using the global locale are subject to the same constraints as with the pre-xlocale libc. Calls to any locale-aware functions in threads using the global locale, while modifying the global locale, have undefined behaviour. Because of this, we have to ensure that we always copy the components of the global locale, rather than alias them. It would be cleaner to simply remove the special treatment of the global locale and have a locale_t lazily allocated for the global context. This would cost a little more `malloc()` memory, so is not done in the initial version. Caching ------- The existing locale implementation included several ad-hoc caching layers. None of these were thread safe. Caching is only really of use for supporting the pattern where the locale is briefly changed to something and then changed back. The current xlocale implementation removes the caching entirely. This pattern is not one that should be encouraged. If you need to make some calls with a modified locale, then you should use the _l suffix versions of the calls, rather than switch the global locale. If you do need to temporarily switch the locale and then switch it back, `uselocale()` provides a way of doing this very easily: It returns the old locale, which can then be passed to a subsequent call to `uselocale()` to restore it, without the need to load any locale data from the disk. If, in the future, it is determined that caching is beneficial, it can be added quite easily in xlocale.c. Given, however, that any locale-aware call is going to be a preparation for presenting data to the user, and so is invariably going to be part of an I/O operation, this seems like a case of premature optimisation. localeconv ---------- The `localeconv()` function is an exception to the immutable-after-creation rule. In the classic implementation, this function returns a pointer to some global storage, which is initialised with the data from the current locale. This is not possible in a multithreaded environment, with multiple locales. Instead, each locale contains a `struct lconv` that is lazily initialised on calls to `localeconv()`. This is not protected by any locking, however this is still safe on any machine where word-sized stores are atomic: two concurrent calls will write the same data into the structure. Explicit Locale Calls --------------------- A large number of functions have been modified to take an explicit `locale_t` parameter. The old APIs are then reimplemented with a call to `__get_locale()` to supply the `locale_t` parameter. This is in line with the Darwin public APIs, but also simplifies the modifications to these functions. The `__get_locale()` function is now the only way to access the current locale within libc. All of the old globals have gone, so there is now a linker error if any functions attempt to use them. The ctype.h functions are a little different. These are not implemented in terms of their locale-aware versions, for performance reasons. Each of these is implemented as a short inline function. Differences to Darwin APIs -------------------------- `strtoq_l()` and `strtouq_l() `are not provided. These are extensions to deprecated functions - we should not be encouraging people to use deprecated interfaces. Locale Placeholders ------------------- The pointer values 0 and -1 have special meanings as `locale_t` values. Any public function that accepts a `locale_t` parameter must use the `FIX_LOCALE()` macro on it before using it. For efficiency, this can be emitted in functions which *only* use their locale parameter as an argument to another public function, as the callee will do the `FIX_LOCALE()` itself. Potential Improvements ---------------------- Currently, the current rune set is accessed via a function call. This makes it fairly expensive to use any of the ctype.h functions. We could improve this quite a lot by storing the rune locale data in a __thread-qualified variable. Several of the existing FreeBSD locale-aware functions appear to be wrong. For example, most of the `strto*()` family should probably use `digittoint_l()`, but instead they assume ASCII. These will break if using a character encoding that does not put numbers and the letters A-F in the same location as ASCII. Some functions, like `strcoll()` only work on single-byte encodings. No attempt has been made to fix existing limitations in the libc functions other than to add support for xlocale. Intuitively, setting a thread-local locale should ensure that all locale-aware functions can be used safely from that thread. In fact, this is not the case in either this implementation or the Darwin one. You must call `duplocale()` or `newlocale()` before calling `uselocale()`. This is a bit ugly, and it would be better if libc ensure that every thread had its own locale object.