Gfyuki

First of all, I am a beginner, so if this question sounds silly, please point out the incorrect assumptions.

From what I understand, the job of an operating system is to manage hardware, and the software that runs on the OS. Also, from what I understand, assembly programs allows one to control hardware almost directly. In an assembly program, one can read and write data into registers, and read and write data into RAM.

Given this freedom to mess with registers and RAM, would it not be possible for assembly programs to affect the operating system? Suppose the OS is using register A to store crucial information. Suppose I write an assembly program, assemble the program, and run the assembled program on the OS. If the program successfully writes junk into register A, the OS will surely be affected.

Questions:

1. Is it possible to mess with register A in the manner described above?




2. If not, what stops assembly programs from modifying the registers used by the OS?

In the end, all programs are machine code, regardless of whether the source language was assembler or a high-level language.

The important thing is that there are hardware mechanisms that limit what a given process can do, including "messing with" registers that could affect other programs or the operating system itself.

This started with a simple distinction between "user" and "supervisor" modes of operation, and has since evolved into a security architecture with multiple "rings" of privilege.

Here's a very generic example to make it a little more concrete:

• In "user mode", a process cannot access memory that hasn't been assigned to its process ID. Memory assigned to other processes and the operating system itself is blocked. This includes the values of registers used by those other processes. This is enforced by the MMU hardware.




• Therefore, in "user mode", a process cannot access the MMU control registers.




• And obviously, in "user mode", a process cannot change the mode to "supervisor mode" except through a very well-defined mechanism that involves calling an operating system function.

The operating system gets control if the process tries to break any of these rules. Once that happens, the OS can simply halt the offending process, never executing any more of its instructions.

Many multitasking operating systems use a data structure called a Process Control Block (PCB) to take care of the register overwrite problem. When you run your code, the OS creates a new process to keep track of it. The PCB contains information about your process and space allocated to hold register contents. Let's say process A is currently running on the processor and your code is in process B. What happens when you run your code goes something lke this:

1. Process A's state data (register contents, program counter, etc.) are copied into its PCB.




2. Process B's state data are copied from its PCB into the CPU registers




3. Process B runs on the processor until it finishes or is preempted




4. Process B's state data are copied back into its PCB




5. Process A's state data are copied back into the CPU and it continues running

If the operating system is running as process A, then you can see how saving its registers before your program runs then copying them back into the CPU once your program ends prevents user programs from messing with what's going in other processes.

To avoid user processes writing over OS data in memory, most platforms use memory segmentation. Basically, using virtual memory, the address space a process sees can be mapped to any arbitrary range of physical addresses. As long as the processes' physical memory spaces don't overlap, it's impossible for one process to overwrite another's data.

Of course, different OSes do things differently, so this won't apply in every case. Also, on most platforms, OS processes run in a different mode from user processes and there are some intricacies there.