[KLUG Hardware] Join "HARDWARE" for "endian/alignment" -- WAS: Calling all Linux novices

[Bryan J. Smith]

--=-Adhmvo2kmI/3kN6LfQBD
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

On Thu, 2003-01-16 at 10:18, Jeremy Leonard wrote:
> Is it possible somebody could explain this endian thing?

First off, for most systems, byte endian isn't as much of an issue as
byte alignment.  But both are still important.

[ NOTE:  If I use a term, like "compiler," that you are not familiar
with, please feel free to ask what it means.  Many people don't know so
don't feel "dumb."  I also added a section called "CISC v. RISC" at the
bottom. ]


- ENDIAN

The term comes from Gulliver's Travels.

"Little endian" architectures put the least significant byte (LSB --
i.e. the smallest part of the value) first, and go up from there.  That
means a number that is 8-bit, 16-bit, 32-bit or any other size will
always have byte 0 (the first byte) start first, then procede with the
next, etc...  So the first two bytes of a 32-bit number match a 16-bit,
etc...

Little endian architectures include Intel x86 and Digital VAX and Alpha.

"Big endian" architectures put the most significant byte (MSB -- i.e.
the biggest part of the value) first, and go down from there.  This
means that 8-bit, 16-bit, 32-bit or any other size will _differ_ in
their bytes.  But it offers other advantages, such as the very first bit
tells if the number of negative or not (two's complement -- long story).

Big endian architectures include IBM 360/370/390 (mainframe) and Power,
Sun SPARC, HP PA-RISC, and Motorola 680x0.  Big endian is also "network
order" -- used for nearly all organization of open standard interfaces
and protocols.

Some architectures are "bi-endian."  They support either.  MIPS, PowerPC
and IA-64 are such.  It doesn't mean the chip is run both ways
simultaneously.  It just means they can use either.

E.g., Windows NT only runs on little endian architectures.  As such, it
was ported from x86 to Alpha (discontinued), MIPS (little endian mode,
discontinued), PowerPC (little endian mode, discontinued) and IA-64
(little endian mode).  MacOS only runs on big endian architectures, so
it was written for 68000 and ported to PowerPC (big endian mode).

GNU is one of the few "bi-endian" platforms, which is why Linux is able
to run on any processor in any support endian mode.

The "problem" with most Windows developers is that they don't consider
the issues involved with endian.  They simply store binary directly to a
file or interface without thought of "network order" or other systems
reading files.  While this is not as critical as "alignment," it is
still important.

Some links ... =20
http://www.cs.umass.edu/~verts/cs32/endian.html =20
http://www.webopedia.com/TERM/b/big_endian.html =20


- ALIGNMENT

Alignment is an even more serious issue from a compatibility standpoint.

Alignment is the concept of organizing code or data on "even
boundaries."  This might be _required_ 4-bytes (32-bit) for most RISC
chips (e.g., it's 64-bit in newer SPARC), or an _optional_ 4-bytes for
an Intel x86 optimizing compiler.  Plus there are optimizations at
"paragraph" (256 bytes for x86) and "page" (4KB for x86) boundaries.

The "requirement" for alignment in most RISC architectures is out of
pure RISC fundamentals.  RISC =3D reduced instruction set computation,
which tries to make instructions less complicated from an "in silicon"
perspective.  By only allowing a "load" on a 32-bit (or other) boundary,
the number of wires required in an ALU or other CPU core decreases by 2
or even 4 fold in the chip.

Now when it comes to most CISC implementations, at least the "CISC
poster child" of Intel x86 (whereas Motorola 68000 CISC actually has
some alignment requirements), it allows you to throw code or data
anywhere you want.  *BUT* by putting the start of code or a value on an
"odd" address, the CISC architecture takes longer to decode and execute
an instruction, reducing performance.  As such, many "optimizing"
compilers align on various boundaries for improved performance.

99% of Windows developers are completely ignorant of byte alignment,
which not only results in compatibility issues between non-x86 systems,
but between changes in Windows compilers.

If a Windows developer creates a data structure, he often does not care
about the actual format of that structure.  Or even if he does take care
to align the data in specific byte boundaries by using the language's
underlying bit/byte options in the structure, they may be incompatible
with a future build.  Why?

Because the newer compiler version might turn on a byte alignment
optimization by default, or change how certain data types are organized
into a structure.  It is _very_important_ that developers who create
complex data structures in the programs _always_ tell the compiler (by
using the syntax of their language) exactly _how_ to organize the
structure physically in-memory and, most importantly, on-disk.

Which is why blindly writing "entire structures" as "binary records" to
a file is the _worst_thing_ you can do.  But basic programming
fundamentals don't point out this fact, and most programmers do this
without thinking.

This includes MS Word for Windows.

As such, when a future version of Visual Studio changes an alignment
option, or optimizes some data organization which results in a different
data alignment in that structure, the structure that is written or read
from a file now changes.  And Microsoft cannot compensate for it because
have _not_ documented how a attribute and values should be organized in
the binary file format!  They blindly just let the compiler decide in MS
Word for Windows.

This can happen even when just creating a "patch" for a version.  Which
explains why some patches "break" things.  Microsoft just doesn't have
the time to check all the things that could break, because they don't
know what to look for in the binary file formatting.

This is quite different than MS Word for Mac, where dealing with
alignment is _not_ optional.  So they design the load file functions in
the Mac version to be more "tolerant" of changes in the binary file
format, looking for specific attributes and _never_ assuming they will
be on a fixed structure boundary.  So files coming from MS Word for
Windows, or already on MS Word for Mac usually load fine.

But when it comes to "nailing down" what the actual binary file format
is for "writing out," they can't because there is no such thing on
Windows.  And most of the time, all they can do is just reuse the same
structure code from the Windows version, and then "guess" on what it
should look like in binary based on analysis of recent MS Word for
Windows binary file.  Because of the load file function changes in MS
Word for Mac, it will be able to read it, but because MS Word for
Windows just uses the same, dumb, blind binary record reads, it will
usually lose a shitload.

Microsoft is currently running into the same issues with porting its
desktop software to the Intel's IA-64 platform, which is an 128-bit VLIW
RISC platform (3 x 41-bit words with 5-bit control).  Hence why
Microsoft is really, really, really hopeful that AMD x86-64 takes over
the desktop market.

-- Bryan

CISC v. RISC

CISC was originally how microprocessors were designed.  A more
"programmer" type defined what instructions (binary opcode+operand --
commonly seen as equivalent "assembler languge") he needed, and the
engineers designed the chip around them, including extensive use of
"microcode."  Microcode is used to break a complex instruction down into
multiple, simplier instructions that can executed in silicon -- which
results in different CISC instructions taking less or more time not just
based on the instruction, but the type of operands (variables) passed.

RISC was an idea that came out of Standford (MIPS) and Berkeley (SPARC)
in the mid-'80s.  By then, most programmers were using higher level
languages, and no longer writing in assembler.  Engineers who designed
the chips now said, "hey, this instruction can only do this before you
start introducing bottlenecks with microcode."  And the compiler now
handled turning programming language code into the unique, less flexible
RISC instructions -- all transparent to the developer (except the
engineers who wrote the compiler ;-).  The concept of RISC is that the
microcode is to be minimized, and most instructions should take the same
amount of time as others.

RISC also tried to "standardize" the size of an instruction to decrease
the decoding and other superscalar logic required.  E.g., x86 CISC
instruction can be anywhere from 8-bit to 144-bit.  Most RISC
instructions are fixed 32-bit.  A simple comparison of the PowerPC
(RISC) versus the AMD Athlon (CISC) shows the complexity of the latter
is twice as much.


--=20
Bryan J. Smith, E.I. (BSECE)       Contact Info:  http://thebs.org
[ http://thebs.org/files/resume/BryanJonSmith_certifications.pdf ]
------------------------------------------------------------------
* A lecture on software piracy from Bill Gates is like a lecture *
* on adultry from the owner of a brothel of other people's wives *


--=-Adhmvo2kmI/3kN6LfQBD
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)

iD8DBQA+Jt5hDjEszaVrzmQRAkQIAKCkTT7Yn/V6x35gqnfW1usLxlTIRwCfRiqf
GMSZiZhHA/q6X1kBzbK/5/M=
=R0eq
-----END PGP SIGNATURE-----

--=-Adhmvo2kmI/3kN6LfQBD--