Rendered at 10:56:34 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
amluto 16 hours ago [-]
This seems weak.
In a language like Rust, the compiler will “lock” the pointers for you, and you can’t forget.
In a language like C++ (and presumably Zig), one could, in theory at least, have the iterators and slices that reference the storage of a dynamic array hold some sort of lock that pins the storage.
But this API requires the programmer to remember to lock the pointers and also requires the programmer to keep the lock alive for the correct region of code. And it looks to me like even the example in the blog post has the lock taken completely outside the function that requires stability, so there is nothing whatsoever that gets the lock scoping right. Even the type system can’t help — the offending parse function can’t declare that it wants a pointer-locked ArrayList parameter.
mirashii 8 hours ago [-]
To make matters worse, there’s also a weaker documentation problem. Where should one learn that they need to do this? zig.guide’s page on ArrayList doesn’t mention it. https://ziglang.org/documentation/master/std/#std.ArrayList doesn’t mention it, https://ziglang.org/documentation/master/std/#std.ArrayList doesn’t mention it at the top level, just a method in the midst of dozens of other methods. I honestly don’t know how one is meant to discover this outside of random blog posts.
csande17 2 hours ago [-]
The devlog is the very first place changes get written about when they land in Zig's master branch (and sometimes even before!), so presumably zig.guide will mention this change once it actually makes it into a release.
“I use it in a lot of places where I know the max capacity ahead of time -- ensureCapacity() followed by a lot of AssumeCapacity()-styled commands. It's convenient for all of the ... convenience ... methods (append() requires some bookkeeping somewhere, appendSlice() requires more, and so on). In those usages, it's basically syntactic sugar over a slice”*
I suspect “where I know the max capacity ahead of time” covers most if not all use cases (if it you use this without knowing max capacity, you either accept your code may panic, or you do some unlock, grow, lock again dance when you discover your initial estimate is wrong)
If so, wouldn’t adding a growable container where you specify capacity at construction time and removing access to the internal pointers of ArrayList be a better way to handle this?
frmdstryr 15 hours ago [-]
Do any languages have a notion of "relative pointers"? So in the example if instead of appending "line" as ptr & len, it'd instead be appending an offset & len which could in theory be used to safely compute the actual location even with relocations.
amluto 14 hours ago [-]
Languages with dependent types can express things like “this offset is in bounds relative to this other array”, which is maybe what you’re thinking of.
sparkie 8 hours ago [-]
I made a mini example in C.
It's awkward to do get right because you need an indirect pointer whose address remains fixed, but points to another pointer which can change (and is volatile).
While it might be possible to make something like this lockless - it's much simpler to stick a mutex in the array header. When we access the array_segment we can take a lock to prevent some other thread reallocating mid-way through accessing.
There's probably a few improvements that could be made. In particular it doesn't handle use-after-free, so it's not thread safe w.r.t cleanup.
That is called an index. If you want it to be standalone, you can bundle it with the ArrayList.
sparkie 11 hours ago [-]
I think parent was after base+offset+index rather than just base+index.
Examples would be eg, `string_view` or `ArraySegment`. They hold some offset relative to a base allocation, and when we index the string_view or ArraySegment we're indexing relative to that offset.
tialaramex 2 hours ago [-]
Presumably `string_view` is referring to C++ std::string_view ?
It's not holding an "offset", it's a fat pointer, (ptr,len) or possibly (start,end)
You're imagining this as (string_ptr,offset,len) but that's 50% bigger for no practical benefit, you cannot unwind a std::string_view to get the string it's a view into, indeed there may never have been such a string.
dminik 11 hours ago [-]
I wish languages made it easier (or possible) to track index ownership at compile time.
afdbcreid 10 hours ago [-]
You can in Rust! You just bundle it with a lifetime.
But... This loses the reason people are using indices to begin with: because the borrow checker cannot track what they do.
Similarly, CPU architectures that use descriptors can (have to?) have languages with that notion.
sparkie 14 hours ago [-]
The FS and GS segment selectors are still used in x86-64, typically for `thread_local` storage, but they can be repurposed.
`thread_local` is an example of a "relative pointer" though. Instructions to access the thread local are prefixed with `fs:` or `gs:`, and point relative to the address in the respective segment register.
A far pointer sounds like the global based pointer described in that article. The far pointer Wikipedia article says they are problematic but doesn't give much reasoning as to why.
sparkie 13 hours ago [-]
Far pointers are for accessing memory in different segments. They're basically obsolete now. They were necessary in older machines with limited sized pointers or address spaces.
GCC still supports `__seg_fs` and `__seg_gs`, which behave similar to `far` in the example on the wiki page, as the FS and GS segment registers are still valid in x86-64 and used for TLS. Clang uses attributes `address_space(257)` and `address_space(256)` for the same thing.
The `__based` pointer in MSVC exploits the addressing modes by pinning the base in eg: `[base+index*scale+displacement]`. It's unrelated to segmentation.
Joker_vD 12 hours ago [-]
> Far pointers are for accessing memory in different segments. They're basically obsolete now.
Project CHERI would like to disagree.
sparkie 11 hours ago [-]
That's Fat pointers, not Far pointers. A fat pointer is a pointer with some other associated data which is stored in the pointer itself - typically by widening the number of bits used to hold a pointer value. The addressable bits usually remain unchanged - the added bits contain the auxiliary data.
Segmentation isn't used. There's no separate registers to hold the bounds information in CHERI - the bounds are held in the pointer value, unlike for example, the now obsolete Intel MPX, which held bounds information in separate registers.
There's some similarity to segmentation because the CHERI pointer restricts which addresses can be accessed, but I wouldn't compare them to far pointers.
Most modern processors have a single linear virtual address space and don't use segmentation, and even where segment registers exist (eg, FS and GS on x86-64), they're only superficial "address spaces" - allocated sections of the process's linear virtual address space which could be accessed without segmentation registers if you knew the base address held in FS or GS.
surajrmal 11 hours ago [-]
Not exactly what you asked, but c++ does this for vtables if you pass the right option to the compiler: -fexperimental-relative-c++-abi-vtables
There is a similar proposal for trait objects in rust.
adzm 11 hours ago [-]
in c++, boost interprocess has offset_ptr which is useful since the shared data structure may be mapped at different locations in memory in each process
geon 12 hours ago [-]
Array indexing?
slopinthebag 13 hours ago [-]
This is how it is with languages which provide less guarantees than Rust. Sure you can try to hold all the invariants and restrictions in your head, but a sufficiently advanced compiler can do this for you without the possibility of making mistakes. I have no idea why people claim that's too restrictive - if you're not enforcing those rules manually you're just setting yourself up for issues down the road.
pron 15 hours ago [-]
I reach for a low-level language only when I want low-level control over what operations happen and when, what memory is used and when etc.. At present, no language offers me this control and safety at the same time. With Rust, when I need such control (which is always, otherwise I would use a higher-level language), I need to give up safety, anyway, at which point I have no safety and the complexity of a language that offers safety.
So right now, when we want control, we need to give up some safety, but weaker things are still helpful.
Also, in low-level code, the problem of "I might forget to do something" sometimes clashes with the problem of "I need to see exactly what operations are done and where". Various kinds of implicitness help with the former at the expense of the latter.
I'm not saying this is universally better than other approaches, but many people who do serious low-level programming would prefer this.
afdbcreid 12 hours ago [-]
> With Rust, when I need such control (which is always, otherwise I would use a higher-level language), I need to give up safety, anyway, at which point I have no safety and the complexity of a language that offers safety.
This is a very, very, very common claim. And unfortunately I have no other way to describe it other than a strawman.
In 95% (at least) of the application that need systems programming (not to talk about all applications that don't necessarily need it but will benefit from the performance and it wasn't an option because C++ wasn't an option), you have at most 20% (wildly overestimating) of code that needs to be unsafe. The rest could be completely safe. And amongst code that must be unsafe, you can very commonly encapsulate it in some safe pattern. Many times even extract it to a reusable crate.
> you have at most 20% (wildly overestimating) of code that needs to be unsafe
Obviously, but that doesn't help me if the complexity in the unsafe parts is made worse, while the safety helps the parts where little help is needed. It's not like the danger in a C program is spread evenly, either.
> but will benefit from the performance
Not so much. Safe Rust is faster than Python and Go for sure, but is, on average, about as fast as Java and C#; sometimes faster, sometimes slower.
weinzierl 36 minutes ago [-]
In addition to that, even unsafe Rust is not like C. It disables a limited set of checks, but Rust's type system, ownership model and bounds checks remain in force.
Not to mention the availability of advanced tooling like MIRI.
yefol 11 hours ago [-]
[flagged]
afdbcreid 10 hours ago [-]
Yes you need to vet touching safe code. Which is why you keep things private, encapsulate them, and extract them into reusable crates.
The most important reason unsafe code is harder to write than C or C++ is that you must keep soundness, something none of these languages have. But yes the different rules also play part (although: do you know a single C or C++ codebase that does not violate TBAA? Some just disable it in the compiler, making them non-standard, while some just leave it potentially exploitable).
But the most important answer is the empirical evidence like I brought above. We have empirical evidence C and C++ codebases cannot be secure. We have empirical evidence Rust codebases can, even with unsafe code. Therefore, Rust is safer, period.
> Do Rust libraries, including std, historically have had UB bugs?
Did C or C++ libraries, historically, have UB bugs? Sorry, that just amplifies the strawman.
> Can Miri catch everything?
Miri is a dynamic analyzer, aka. a sanitizer. It will catch anything you test. It's like in C and C++, except you only need it for unsafe code.
> Are all the rules of unsafe, pinning, etc. fully specified and easy to learn and reason about?
Fully specified? People are working on it (are C's and C++'s UB rules fully specified? I'll save you the answer: no. Yes there is a standard and it's woefully incomplete).
Easy to learn and reason about? Probably not, which is why not everyone should be writing unsafe code.
Possible to learn and reason about? Absolutely yes. Especially with existing and emerging dynamic and static analyzers.
dwattttt 10 hours ago [-]
I'm not super certain you're interested in answers, but assuming good faith:
The claim isn't "there's no unsafe". You've linked one file out of an entire stdlib; it uses unsafe to implement its algorithm, and of all the Rust code that could exist, this has one of the highest requirements for being maximally performant.
Now if you'd said "most of the Rust std library is unsafe", or "most Rust code is unsafe, you'd have a good rebuttal. But that's not the case.
> And, if you have an unsafe block that is 100% correct, but it relies on safe code being correct, do you need to vet all that safe code? Potentially whole modules needing to be vetted?
Then the unsafe block is not 100% correct. I can slap a wrapper around memcpy and call it "safe", and say that if anyone passes wrong parameters it's their fault. Rust as a language says I'm at fault for saying it's safe though.
> Is unsafe Rust code generally harder to get correct than code in other languages, due to...
Harder than other systems programming languages? Having worked in a fair few, I disagree. Harder than "higher" level languages? Some of them yes, some of them no; I've seen "simple" languages admit very poor architectures, and fall in a "safe" heap when the project has to grow.
Are you suggesting this is a bar a language should achieve? Some examples of this would be interesting.
As for the rest, I don't think anything meets this bar you're setting. Certainly not languages that would otherwise be used where Rust is.
surajrmal 8 hours ago [-]
I've written systems level code (drivers and os code) for years and outside of ffi, I've managed to go on year long stretches without touching unsafe. It's really not a commonly needed tool in a well architected code base with good libraries to encapsulate common reasons it might otherwise be necessary. And we don't really consider using unsafe taboo, it's just not necessary.
xeonmc 14 hours ago [-]
Those who would give up low-level control to purchase a little memory safety, deserve neither control nor safety.”
- Benjamin Franklin, or something like that
duped 15 hours ago [-]
What are some examples of things you "always" need that require unsafe Rust?
delamon 47 minutes ago [-]
Objects belonging to multiple double-linked lists at the same time. Easily done with intrusive lists. Safe rust would require Rc/Arc: penalty both on memory usage and cpu time.
yefol 11 hours ago [-]
Not him, but projects that need performance often use unsafe or otherwise allow for UB. Embedded is arguably another example, since no_std allows UB even without unsafe, for instance by causing a stack overflow.
afdbcreid 9 hours ago [-]
> for instance by causing a stack overflow
That's not "for instance", that's literally the only place Rust has unfixable UB on embedded (code on OS has other such things, e.g. reading/writing to `/proc/self/mem`).
> projects that need performance often use unsafe
You'll be surprised to hear how often it's not needed at all. And when it is, you'll be surprised to hear how many times you can still avoid it with some tricks. Contrary to popular belief, performance isn't the most common reason for unsafe (FFI probably is).
insanitybit 9 hours ago [-]
I haven't needed `unsafe` for performance since crates like zerocopy etc exist. It's been years, and I've worked hard to shave nanoseconds off of code, using valgrind to measure single digit changes to branch predictions.
pjmlp 15 hours ago [-]
Except the point that Zig should do better than Object Pascal, Modula-2, with solutions already available on Insure++ and friends for use after free, 30 years ago.
slopinthebag 13 hours ago [-]
But the point of unsafe {} in Rust is not that you should never use it, it's that it creates a clear boundary between code that is safe and the code that needs that lower level control. In other languages, everything is inside an unsafe block. If everything you do requires such low level control over every allocation and access, it sounds like you should be using assembly.
insanitybit 9 hours ago [-]
2026 and developers still use memory unsafe languages. I hope we get regulated at this point, disgusting.
_bohm 18 hours ago [-]
It's a nice feature but I can't help feeling like, if you need a stable pointer to an item in a collection, ArrayList is the wrong data structure to use? Maybe someone can chime in and give me an example of when you'd do this instead of, e.g., just storing an index. Alternatively, you could use an Unrolled Linked List (FKA SegmentedList in Zig before it was removed in 0.16, not sure why).
ivanjermakov 13 hours ago [-]
I count this as a "rookie at system programming" mistake alongside returning a reference to a local variable. Rust is great at this because borrow checker can catch those at compile time and it can _teach_ devs to not do that.
dataflow 16 hours ago [-]
I don't know Zig, but conceptually: a direct pointer is the fastest way to access an object. An arraylist is the fastest dynamic sequence of objects (fattest in access, not in growth). You use these when you need the performance. It's not often but it certainly happens. The most trivial example is a string that you append to but still need to pass to a C API in between that expects it to be contagious, but it's far more useful than just for storing characters.
kllrnohj 11 hours ago [-]
Indexing into an array is direct pointer access, there's just an addition in front of it but it's hard to imagine that showing up at all in even the tightest of benchmark loops
dataflow 8 hours ago [-]
I can't speak for your imagination, but this absolutely does come up if you're writing high-performance code.
Also note that being able to access arbitrary objects (as opposed to objects in the same array) requires storing two pieces of information: an index and a pointer to the beginning of the array. So it can use twice as much memory, which affects your cache etc., though you don't even need that to see the effect.
_bohm 15 hours ago [-]
Sure, in cases where you need elements to be contiguous in memory then certainly an unrolled linked list is not appropriate. There's usually not a meaningful performance difference between a pointer deref and an indexed array access, however.
hansvm 16 hours ago [-]
I use it in a lot of places where I know the max capacity ahead of time -- ensureCapacity() followed by a lot of *AssumeCapacity()-styled commands. It's convenient for all of the ... convenience ... methods (append() requires some bookkeeping somewhere, appendSlice() requires more, and so on). In those usages, it's basically syntactic sugar over a slice. That's not a perfect solution, but it's reasonably good often enough that I keep doing it.
The proposed change doesn't do much for me personally (memory safety is ensured in other ways, and if it weren't I wouldn't be annoyed debugging the allocator-observed errors), but I could see myself using it at some other point in time for the same class of usages, or I could see other people relying on it when they choose that class of coding.
nvme0n1p1 17 hours ago [-]
SegmentedList had a weird API, especially the way you control the list growth factor by the size of an inline array. And it hadn't kept up with stdlib norms in recent versions. I do hope it comes back eventually with an improved API.
At least we got Deque in exchange. I use that far more often than I used SegmentedList.
_bohm 17 hours ago [-]
Aha, gotcha. To tell you the truth, I don't think I ever used it. I have a custom implementation I wrote because I didn't realize at the time that SegmentedList was an unrolled linked list :p
bvrmn 16 hours ago [-]
ArrayList is a very generic (pun not intended) structure and could be stretched quite freely in any direction with useful property of owning underlying slice. Like readonly preallocated ArrayList is a thing.
_bohm 16 hours ago [-]
That's a good point. Using lockPointers would be a good way to enforce at runtime that your ArrayList is truly read-only.
the__alchemist 16 hours ago [-]
My mental model of Zig is that it is explicitly the language for developers who prefer using pointers in business logic (instead of just in MMIMO, and are looking for something with improvements over C); i.e. exactly this class of abstraction.
_bohm 16 hours ago [-]
Having written a bunch of Zig, I wouldn't say that the language design or culture explicitly encourages the use of pointers over indices in such situations. I would say it's more a language which trusts the programmer to make correct decisions about which constructs are appropriate in any given circumstance.
the__alchemist 3 hours ago [-]
I appreciate the explanation! I'm a noob at this sort of thing. (My experience with pointers is limited to MMIO and C FFI).
Regarding trusting the programmer: is that in the context of pointers, or more broadly? I'm betting I'm missing more, e.g. maybe mutation control too across threads/cores etc? Or as a broad design principle? (I have written some Zig as a learning dive, but don't Grok it, as am waiting for some core functionality like HALs, GUI libs, 3D libs etc. Was also a bit miffed by the 'operator overloading will never be allowed' as the use cases where I use low level languages have a near total overlap with the ones where I use vectors, quaternions, and matrices.
Stated another way: I learned with higher level langs first, which I believe biased my mental model. Example: Say I want to use a C library in my rust program/lib. (Example: CMSIS DSP). One of the actions I take in the wrapper is convert the pointer to an array ref; my mental model is the param is a list of items; it is divorced from memory. If I want to read/write a reg, or access FLASH, that's where I look to pointers. I e I think zig vs others is about if you want to conflate or divorce collections and memory.
svachalek 16 hours ago [-]
Hasn't the past 30 years of the Internet age taught us that given such trust, programmers will make the incorrect decision with horrifying predictability? The most trivial level of software security requires that pointer safety needs to be mathematically proven not up to human (or LLM) judgment.
yefol 11 hours ago [-]
Does that include deadlock safety?
Try running both of these examples. They only differ in a pair of curly braces.
Who cares what mojo handles? This is about Zig and memory safety.
insanitybit 9 hours ago [-]
Yes, undoubtedly. Anyone in denial of this should be legally barred from programming.
beepbooptheory 17 hours ago [-]
Maybe one use case is if you are interfacing with external C library and you're stuck with pointers?
afdbcreid 15 hours ago [-]
They forgot to add (I believe this was not deliberate, maybe their users already infer that) that this only actually performs the check on Debug and ReleaseSafe modes, not on ReleaseFast mode. Which is reasonable I guess because this is a memory write/read/branch in a super hot code path, but undermines a large part of the guarantee in my opinion (doesn't Zig have a debug allocator that could catch the mistake in the example just as well?).
ivanjermakov 13 hours ago [-]
Debug allocator can't catch it because it's not an allocation bug. Debug allocator finds bugs by marking memory during alloc/free and inspects them upon deinit. Pointer to a memory location change is not something allocator has control over. Possible solutions: smart array list implementation (this article), move semantic analysis (Rust's borrow checker), runtime introspection (https://fil-c.org/).
afdbcreid 12 hours ago [-]
It is an allocation bug, it's a use-after-free. It's only a UAF if you actually have a reallocation (something that happens in the given example), but debug allocators don't require you to annotate your code.
portly 14 hours ago [-]
This makes a lot of sense if you consider that it is consistent with the rest of the language. It is one more way to set up tripwires in your code to to catch your own programming errors. Similar to using asserts in your functions to vet input and output.
I use Array list a lot so excited to add this throughout the code to harden them.
I can imagine this is not everyone's cup of tea, but then you probably also wouldn't enjoy any of the other explicitness.
Rendello 16 hours ago [-]
Aside: one Zig (syntax) feature that I really missed in Rust is shown in the second code block, namely prefixed multi-line string literals à la:
const text =
\\This is a long comment
\\But I can split it among lines arbitrarily
\\And keep my indentation.
;
I've started using the Rust macro library `docstr` [1], which does the same thing:
const TEXT: &'static str = docstr!(
/// Now I can do it in Rust, too.
/// I prefer this style a lot of the time
/// for long texts.
);
It even works with macros (example from the docs):
let greeting: String = docstr!(format!
/// Hello, my name is {name}.
/// I am {} years old!
age
);
C++ developers constantly wrestle with `vector` iterator invalidations. Good on Zig for making this a first-class concern.
brcmthrowaway 10 hours ago [-]
How does Rust avoid this?
trucks-refinish 10 hours ago [-]
In rust you cannot mutate anything that is being read basically. The borrow checker enforces this.
So an iterator takes an immutable reference to the vector and mutation requires a mut ref, and you can't have both at the same time.
boricj 15 hours ago [-]
It took me a minute to understand that this asserts on pointer change within the container, rather than lock/unlock the data structure like a SDL surface.
I recently implemented a custom C++ container for a path whose components could be iterated, backed by a std::string. I just store indices and a reference to the string, such that my iterators are not invalidated if the std::string gets reallocated after being modified. Far less error prone for little added cost.
Prydown 11 hours ago [-]
Constant headache in C++ with `std::vector` element references. Zig's explicit stability here is a welcome relief for data structures.
afdbcreid 9 hours ago [-]
It seems you misunderstood. It's not an explicit stability guarantee (such things is not possible), it's a debugging helper to crash the program more easily when it happens, requiring you to annotate the code.
duped 15 hours ago [-]
This is a gripe of mine, and I will admit it is weak.
Changing a segfault to a panic with a stack trace is an improvement in developer experience. It does not make better software. The advantage of automatic strategies to mitigate memory safety mistakes either by using GC to make the program sound or static analysis to prevent the mistake by construction is plainly better.
There is a direction in some systems programming circles away from this by eschewing "complexity" (in other words, fixing the damn problems) for programs that have better error messages when the programmer made a mistake. I don't see that as better software.
In a language like Rust, the compiler will “lock” the pointers for you, and you can’t forget.
In a language like C++ (and presumably Zig), one could, in theory at least, have the iterators and slices that reference the storage of a dynamic array hold some sort of lock that pins the storage.
But this API requires the programmer to remember to lock the pointers and also requires the programmer to keep the lock alive for the correct region of code. And it looks to me like even the example in the blog post has the lock taken completely outside the function that requires stability, so there is nothing whatsoever that gets the lock scoping right. Even the type system can’t help — the offending parse function can’t declare that it wants a pointer-locked ArrayList parameter.
“I use it in a lot of places where I know the max capacity ahead of time -- ensureCapacity() followed by a lot of AssumeCapacity()-styled commands. It's convenient for all of the ... convenience ... methods (append() requires some bookkeeping somewhere, appendSlice() requires more, and so on). In those usages, it's basically syntactic sugar over a slice”*
I suspect “where I know the max capacity ahead of time” covers most if not all use cases (if it you use this without knowing max capacity, you either accept your code may panic, or you do some unlock, grow, lock again dance when you discover your initial estimate is wrong)
If so, wouldn’t adding a growable container where you specify capacity at construction time and removing access to the internal pointers of ArrayList be a better way to handle this?
It's awkward to do get right because you need an indirect pointer whose address remains fixed, but points to another pointer which can change (and is volatile).
While it might be possible to make something like this lockless - it's much simpler to stick a mutex in the array header. When we access the array_segment we can take a lock to prevent some other thread reallocating mid-way through accessing.
There's probably a few improvements that could be made. In particular it doesn't handle use-after-free, so it's not thread safe w.r.t cleanup.
https://godbolt.org/z/rYzn5KGre
Examples would be eg, `string_view` or `ArraySegment`. They hold some offset relative to a base allocation, and when we index the string_view or ArraySegment we're indexing relative to that offset.
It's not holding an "offset", it's a fat pointer, (ptr,len) or possibly (start,end)
You're imagining this as (string_ptr,offset,len) but that's 50% bigger for no practical benefit, you cannot unwind a std::string_view to get the string it's a view into, indeed there may never have been such a string.
But... This loses the reason people are using indices to begin with: because the borrow checker cannot track what they do.
Similarly, CPU architectures that use descriptors can (have to?) have languages with that notion.
`thread_local` is an example of a "relative pointer" though. Instructions to access the thread local are prefixed with `fs:` or `gs:`, and point relative to the address in the respective segment register.
A far pointer sounds like the global based pointer described in that article. The far pointer Wikipedia article says they are problematic but doesn't give much reasoning as to why.
GCC still supports `__seg_fs` and `__seg_gs`, which behave similar to `far` in the example on the wiki page, as the FS and GS segment registers are still valid in x86-64 and used for TLS. Clang uses attributes `address_space(257)` and `address_space(256)` for the same thing.
The `__based` pointer in MSVC exploits the addressing modes by pinning the base in eg: `[base+index*scale+displacement]`. It's unrelated to segmentation.
Project CHERI would like to disagree.
Segmentation isn't used. There's no separate registers to hold the bounds information in CHERI - the bounds are held in the pointer value, unlike for example, the now obsolete Intel MPX, which held bounds information in separate registers.
There's some similarity to segmentation because the CHERI pointer restricts which addresses can be accessed, but I wouldn't compare them to far pointers.
Most modern processors have a single linear virtual address space and don't use segmentation, and even where segment registers exist (eg, FS and GS on x86-64), they're only superficial "address spaces" - allocated sections of the process's linear virtual address space which could be accessed without segmentation registers if you knew the base address held in FS or GS.
There is a similar proposal for trait objects in rust.
So right now, when we want control, we need to give up some safety, but weaker things are still helpful.
Also, in low-level code, the problem of "I might forget to do something" sometimes clashes with the problem of "I need to see exactly what operations are done and where". Various kinds of implicitness help with the former at the expense of the latter.
I'm not saying this is universally better than other approaches, but many people who do serious low-level programming would prefer this.
This is a very, very, very common claim. And unfortunately I have no other way to describe it other than a strawman.
In 95% (at least) of the application that need systems programming (not to talk about all applications that don't necessarily need it but will benefit from the performance and it wasn't an option because C++ wasn't an option), you have at most 20% (wildly overestimating) of code that needs to be unsafe. The rest could be completely safe. And amongst code that must be unsafe, you can very commonly encapsulate it in some safe pattern. Many times even extract it to a reusable crate.
That is the point of Rust. Not avoiding unsafety, but limiting and encapsulating it. And evidence proves that to work (for example https://blog.google/security/rust-in-android-move-fast-fix-t...).
Obviously, but that doesn't help me if the complexity in the unsafe parts is made worse, while the safety helps the parts where little help is needed. It's not like the danger in a C program is spread evenly, either.
> but will benefit from the performance
Not so much. Safe Rust is faster than Python and Go for sure, but is, on average, about as fast as Java and C#; sometimes faster, sometimes slower.
Not to mention the availability of advanced tooling like MIRI.
The most important reason unsafe code is harder to write than C or C++ is that you must keep soundness, something none of these languages have. But yes the different rules also play part (although: do you know a single C or C++ codebase that does not violate TBAA? Some just disable it in the compiler, making them non-standard, while some just leave it potentially exploitable).
But the most important answer is the empirical evidence like I brought above. We have empirical evidence C and C++ codebases cannot be secure. We have empirical evidence Rust codebases can, even with unsafe code. Therefore, Rust is safer, period.
> Do Rust libraries, including std, historically have had UB bugs?
Did C or C++ libraries, historically, have UB bugs? Sorry, that just amplifies the strawman.
> Can Miri catch everything?
Miri is a dynamic analyzer, aka. a sanitizer. It will catch anything you test. It's like in C and C++, except you only need it for unsafe code.
> Are all the rules of unsafe, pinning, etc. fully specified and easy to learn and reason about?
Fully specified? People are working on it (are C's and C++'s UB rules fully specified? I'll save you the answer: no. Yes there is a standard and it's woefully incomplete).
Easy to learn and reason about? Probably not, which is why not everyone should be writing unsafe code.
Possible to learn and reason about? Absolutely yes. Especially with existing and emerging dynamic and static analyzers.
> https://github.com/rust-lang/rust/blob/main/library/core/src... How large a percentage of the logic code there is inside of an unsafe block?
The claim isn't "there's no unsafe". You've linked one file out of an entire stdlib; it uses unsafe to implement its algorithm, and of all the Rust code that could exist, this has one of the highest requirements for being maximally performant.
Now if you'd said "most of the Rust std library is unsafe", or "most Rust code is unsafe, you'd have a good rebuttal. But that's not the case.
> And, if you have an unsafe block that is 100% correct, but it relies on safe code being correct, do you need to vet all that safe code? Potentially whole modules needing to be vetted?
Then the unsafe block is not 100% correct. I can slap a wrapper around memcpy and call it "safe", and say that if anyone passes wrong parameters it's their fault. Rust as a language says I'm at fault for saying it's safe though.
> Is unsafe Rust code generally harder to get correct than code in other languages, due to...
Harder than other systems programming languages? Having worked in a fair few, I disagree. Harder than "higher" level languages? Some of them yes, some of them no; I've seen "simple" languages admit very poor architectures, and fall in a "safe" heap when the project has to grow.
> Do Rust libraries, including std, historically have had UB bugs? https://materialize.com/blog/rust-concurrency-bug-unbounded-...
Are you suggesting this is a bar a language should achieve? Some examples of this would be interesting.
As for the rest, I don't think anything meets this bar you're setting. Certainly not languages that would otherwise be used where Rust is.
That's not "for instance", that's literally the only place Rust has unfixable UB on embedded (code on OS has other such things, e.g. reading/writing to `/proc/self/mem`).
> projects that need performance often use unsafe
You'll be surprised to hear how often it's not needed at all. And when it is, you'll be surprised to hear how many times you can still avoid it with some tricks. Contrary to popular belief, performance isn't the most common reason for unsafe (FFI probably is).
Also note that being able to access arbitrary objects (as opposed to objects in the same array) requires storing two pieces of information: an index and a pointer to the beginning of the array. So it can use twice as much memory, which affects your cache etc., though you don't even need that to see the effect.
The proposed change doesn't do much for me personally (memory safety is ensured in other ways, and if it weren't I wouldn't be annoyed debugging the allocator-observed errors), but I could see myself using it at some other point in time for the same class of usages, or I could see other people relying on it when they choose that class of coding.
At least we got Deque in exchange. I use that far more often than I used SegmentedList.
Regarding trusting the programmer: is that in the context of pointers, or more broadly? I'm betting I'm missing more, e.g. maybe mutation control too across threads/cores etc? Or as a broad design principle? (I have written some Zig as a learning dive, but don't Grok it, as am waiting for some core functionality like HALs, GUI libs, 3D libs etc. Was also a bit miffed by the 'operator overloading will never be allowed' as the use cases where I use low level languages have a near total overlap with the ones where I use vectors, quaternions, and matrices.
Stated another way: I learned with higher level langs first, which I believe biased my mental model. Example: Say I want to use a C library in my rust program/lib. (Example: CMSIS DSP). One of the actions I take in the wrapper is convert the pointer to an array ref; my mental model is the param is a list of items; it is divorced from memory. If I want to read/write a reg, or access FLASH, that's where I look to pointers. I e I think zig vs others is about if you want to conflate or divorce collections and memory.
Try running both of these examples. They only differ in a pair of curly braces.
https://play.rust-lang.org/?version=stable&mode=debug&editio...
https://play.rust-lang.org/?version=stable&mode=debug&editio...
https://fasterthanli.me/articles/a-rust-match-made-in-hell
Mojo handles this significantly better than Rust.
I use Array list a lot so excited to add this throughout the code to harden them.
I can imagine this is not everyone's cup of tea, but then you probably also wouldn't enjoy any of the other explicitness.
So an iterator takes an immutable reference to the vector and mutation requires a mut ref, and you can't have both at the same time.
I recently implemented a custom C++ container for a path whose components could be iterated, backed by a std::string. I just store indices and a reference to the string, such that my iterators are not invalidated if the std::string gets reallocated after being modified. Far less error prone for little added cost.
Changing a segfault to a panic with a stack trace is an improvement in developer experience. It does not make better software. The advantage of automatic strategies to mitigate memory safety mistakes either by using GC to make the program sound or static analysis to prevent the mistake by construction is plainly better.
There is a direction in some systems programming circles away from this by eschewing "complexity" (in other words, fixing the damn problems) for programs that have better error messages when the programmer made a mistake. I don't see that as better software.