USIII and beyond CPUs have stricter requirements when it comes

to synchronization needed after stores to internal ASIs in order
to make side-effects visible. This mainly requires the MEMBAR #Sync
after such stores to be replaced with a FLUSH. We use KERNBASE as
the address to FLUSH as it is guaranteed to not trap. Actually,
the USII synchronization rules also already require a FLUSH in
pretty much all of the cases changed.
We're also hitting an additional USIII synchronization rule which
requires stores to AA_IMMU_SFSR to be immediately followed by a DONE,
FLUSH or RETRY. Doing so triggers a RED state exception though so
leave the MEMBAR #Sync. Linux apparently also has gotten away with
doing the same for quite some time now, apart from the fact that
it's not clear to me why we need to clear the valid bit from the
SFSR in the first place.

Reviewed by:	nwhitehorn

1 files changed, 2 insertions, 1 deletions

diff --git a/sys/boot/sparc64/loader/main.c b/sys/boot/sparc64/loader/main.c
index 34fe4ce..5091caa 100644
--- a/sys/boot/sparc64/loader/main.c
+++ b/sys/boot/sparc64/loader/main.c
@@ -37,6 +37,7 @@ __FBSDID("$FreeBSD$");
 #include <machine/tlb.h>
 #include <machine/upa.h>
 #include <machine/ver.h>
+#include <machine/vmparam.h>
 
 #include "bootstrap.h"
 #include "libofw.h"
@@ -461,7 +462,7 @@ itlb_enter_sun4u(u_long vpn, u_long data)
 	stxa(AA_IMMU_TAR, ASI_IMMU,
 	     TLB_TAR_VA(vpn) | TLB_TAR_CTX(TLB_CTX_KERNEL));
 	stxa(0, ASI_ITLB_DATA_IN_REG, data);
-	membar(Sync);
+	flush(KERNBASE);
 	wrpr(pstate, reg, 0);
 }