mirror of
https://github.com/superseriousbusiness/gotosocial.git
synced 2024-12-05 18:12:46 +00:00
1588 lines
84 KiB
Markdown
1588 lines
84 KiB
Markdown
|
# Notable rationale of wazero
|
|||
|
|
|||
|
## Zero dependencies
|
|||
|
|
|||
|
Wazero has zero dependencies to differentiate itself from other runtimes which
|
|||
|
have heavy impact usually due to CGO. By avoiding CGO, wazero avoids
|
|||
|
prerequisites such as shared libraries or libc, and lets users keep features
|
|||
|
like cross compilation.
|
|||
|
|
|||
|
Avoiding go.mod dependencies reduces interference on Go version support, and
|
|||
|
size of a statically compiled binary. However, doing so brings some
|
|||
|
responsibility into the project.
|
|||
|
|
|||
|
Go's native platform support is good: We don't need platform-specific code to
|
|||
|
get monotonic time, nor do we need much work to implement certain features
|
|||
|
needed by our compiler such as `mmap`. That said, Go does not support all
|
|||
|
common operating systems to the same degree. For example, Go 1.18 includes
|
|||
|
`Mprotect` on Linux and Darwin, but not FreeBSD.
|
|||
|
|
|||
|
The general tradeoff the project takes from a zero dependency policy is more
|
|||
|
explicit support of platforms (in the compiler runtime), as well a larger and
|
|||
|
more technically difficult codebase.
|
|||
|
|
|||
|
At some point, we may allow extensions to supply their own platform-specific
|
|||
|
hooks. Until then, one end user impact/tradeoff is some glitches trying
|
|||
|
untested platforms (with the Compiler runtime).
|
|||
|
|
|||
|
### Why do we use CGO to implement system calls on darwin?
|
|||
|
|
|||
|
wazero is dependency and CGO free by design. In some cases, we have code that
|
|||
|
can optionally use CGO, but retain a fallback for when that's disabled. The only
|
|||
|
operating system (`GOOS`) we use CGO by default in is `darwin`.
|
|||
|
|
|||
|
Unlike other operating systems, regardless of `CGO_ENABLED`, Go always uses
|
|||
|
"CGO" mechanisms in the runtime layer of `darwin`. This is explained in
|
|||
|
[Statically linked binaries on Mac OS X](https://developer.apple.com/library/archive/qa/qa1118/_index.html#//apple_ref/doc/uid/DTS10001666):
|
|||
|
|
|||
|
> Apple does not support statically linked binaries on Mac OS X. A statically
|
|||
|
> linked binary assumes binary compatibility at the kernel system call
|
|||
|
> interface, and we do not make any guarantees on that front. Rather, we strive
|
|||
|
> to ensure binary compatibility in each dynamically linked system library and
|
|||
|
> framework.
|
|||
|
|
|||
|
This plays to our advantage for system calls that aren't yet exposed in the Go
|
|||
|
standard library, notably `futimens` for nanosecond-precision timestamp
|
|||
|
manipulation.
|
|||
|
|
|||
|
### Why not x/sys
|
|||
|
|
|||
|
Going beyond Go's SDK limitations can be accomplished with their [x/sys library](https://pkg.go.dev/golang.org/x/sys/unix).
|
|||
|
For example, this includes `zsyscall_freebsd_amd64.go` missing from the Go SDK.
|
|||
|
|
|||
|
However, like all dependencies, x/sys is a source of conflict. For example,
|
|||
|
x/sys had to be in order to upgrade to Go 1.18.
|
|||
|
|
|||
|
If we depended on x/sys, we could get more precise functionality needed for
|
|||
|
features such as clocks or more platform support for the compiler runtime.
|
|||
|
|
|||
|
That said, formally supporting an operating system may still require testing as
|
|||
|
even use of x/sys can require platform-specifics. For example, [mmap-go](https://github.com/edsrzf/mmap-go)
|
|||
|
uses x/sys, but also mentions limitations, some not surmountable with x/sys
|
|||
|
alone.
|
|||
|
|
|||
|
Regardless, we may at some point introduce a separate go.mod for users to use
|
|||
|
x/sys as a platform plugin without forcing all users to maintain that
|
|||
|
dependency.
|
|||
|
|
|||
|
## Project structure
|
|||
|
|
|||
|
wazero uses internal packages extensively to balance API compatibility desires for end users with the need to safely
|
|||
|
share internals between compilers.
|
|||
|
|
|||
|
End-user packages include `wazero`, with `Config` structs, `api`, with shared types, and the built-in `wasi` library.
|
|||
|
Everything else is internal.
|
|||
|
|
|||
|
We put the main program for wazero into a directory of the same name to match conventions used in `go install`,
|
|||
|
notably the name of the folder becomes the binary name. We chose to use `cmd/wazero` as it is common practice
|
|||
|
and less surprising than `wazero/wazero`.
|
|||
|
|
|||
|
### Internal packages
|
|||
|
|
|||
|
Most code in wazero is internal, and it is acknowledged that this prevents external implementation of facets such as
|
|||
|
compilers or decoding. It also prevents splitting this code into separate repositories, resulting in a larger monorepo.
|
|||
|
This also adds work as more code needs to be centrally reviewed.
|
|||
|
|
|||
|
However, the alternative is neither secure nor viable. To allow external implementation would require exporting symbols
|
|||
|
public, such as the `CodeSection`, which can easily create bugs. Moreover, there's a high drift risk for any attempt at
|
|||
|
external implementations, compounded not just by wazero's code organization, but also the fast moving Wasm and WASI
|
|||
|
specifications.
|
|||
|
|
|||
|
For example, implementing a compiler correctly requires expertise in Wasm, Golang and assembly. This requires deep
|
|||
|
insight into how internals are meant to be structured and the various tiers of testing required for `wazero` to result
|
|||
|
in a high quality experience. Even if someone had these skills, supporting external code would introduce variables which
|
|||
|
are constants in the central one. Supporting an external codebase is harder on the project team, and could starve time
|
|||
|
from the already large burden on the central codebase.
|
|||
|
|
|||
|
The tradeoffs of internal packages are a larger codebase and responsibility to implement all standard features. It also
|
|||
|
implies thinking about extension more as forking is not viable for reasons above also. The primary mitigation of these
|
|||
|
realities are friendly OSS licensing, high rigor and a collaborative spirit which aim to make contribution in the shared
|
|||
|
codebase productive.
|
|||
|
|
|||
|
### Avoiding cyclic dependencies
|
|||
|
|
|||
|
wazero shares constants and interfaces with internal code by a sharing pattern described below:
|
|||
|
* shared interfaces and constants go in one package under root: `api`.
|
|||
|
* user APIs and structs depend on `api` and go into the root package `wazero`.
|
|||
|
* e.g. `InstantiateModule` -> `/wasm.go` depends on the type `api.Module`.
|
|||
|
* implementation code can also depend on `api` in a corresponding package under `/internal`.
|
|||
|
* Ex package `wasm` -> `/internal/wasm/*.go` and can depend on the type `api.Module`.
|
|||
|
|
|||
|
The above guarantees no cyclic dependencies at the cost of having to re-define symbols that exist in both packages.
|
|||
|
For example, if `wasm.Store` is a type the user needs access to, it is narrowed by a cover type in the `wazero`:
|
|||
|
|
|||
|
```go
|
|||
|
type runtime struct {
|
|||
|
s *wasm.Store
|
|||
|
}
|
|||
|
```
|
|||
|
|
|||
|
This is not as bad as it sounds as mutations are only available via configuration. This means exported functions are
|
|||
|
limited to only a few functions.
|
|||
|
|
|||
|
### Avoiding security bugs
|
|||
|
|
|||
|
In order to avoid security flaws such as code insertion, nothing in the public API is permitted to write directly to any
|
|||
|
mutable symbol in the internal package. For example, the package `api` is shared with internal code. To ensure
|
|||
|
immutability, the `api` package cannot contain any mutable public symbol, such as a slice or a struct with an exported
|
|||
|
field.
|
|||
|
|
|||
|
In practice, this means shared functionality like memory mutation need to be implemented by interfaces.
|
|||
|
|
|||
|
Here are some examples:
|
|||
|
* `api.Memory` protects access by exposing functions like `WriteFloat64Le` instead of exporting a buffer (`[]byte`).
|
|||
|
* There is no exported symbol for the `[]byte` representing the `CodeSection`
|
|||
|
|
|||
|
Besides security, this practice prevents other bugs and allows centralization of validation logic such as decoding Wasm.
|
|||
|
|
|||
|
## API Design
|
|||
|
|
|||
|
### Why is `context.Context` inconsistent?
|
|||
|
|
|||
|
It may seem strange that only certain API have an initial `context.Context`
|
|||
|
parameter. We originally had a `context.Context` for anything that might be
|
|||
|
traced, but it turned out to be only useful for lifecycle and host functions.
|
|||
|
|
|||
|
For instruction-scoped aspects like memory updates, a context parameter is too
|
|||
|
fine-grained and also invisible in practice. For example, most users will use
|
|||
|
the compiler engine, and its memory, global or table access will never use go's
|
|||
|
context.
|
|||
|
|
|||
|
### Why does `api.ValueType` map to uint64?
|
|||
|
|
|||
|
WebAssembly allows functions to be defined either by the guest or the host,
|
|||
|
with signatures expressed as WebAssembly types. For example, `i32` is a 32-bit
|
|||
|
type which might be interpreted as signed. Function signatures can have zero or
|
|||
|
more parameters or results even if WebAssembly 1.0 allows up to one result.
|
|||
|
|
|||
|
The guest can export functions, so that the host can call it. In the case of
|
|||
|
wazero, the host is Go and an exported function can be called via
|
|||
|
`api.Function`. `api.Function` allows users to supply parameters and read
|
|||
|
results as a slice of uint64. For example, if there are no results, an empty
|
|||
|
slice is returned. The user can learn the signature via `FunctionDescription`,
|
|||
|
which returns the `api.ValueType` corresponding to each parameter or result.
|
|||
|
`api.ValueType` defines the mapping of WebAssembly types to `uint64` values for
|
|||
|
reason described in this section. The special case of `v128` is also mentioned
|
|||
|
below.
|
|||
|
|
|||
|
wazero maps each value type to a uint64 values because it holds the largest
|
|||
|
type in WebAssembly 1.0 (i64). A slice allows you to express empty (e.g. a
|
|||
|
nullary signature), for example a start function.
|
|||
|
|
|||
|
Here's an example of calling a function, noting this syntax works for both a
|
|||
|
signature `(param i32 i32) (result i32)` and `(param i64 i64) (result i64)`
|
|||
|
```go
|
|||
|
x, y := uint64(1), uint64(2)
|
|||
|
results, err := mod.ExportedFunction("add").Call(ctx, x, y)
|
|||
|
if err != nil {
|
|||
|
log.Panicln(err)
|
|||
|
}
|
|||
|
fmt.Printf("%d + %d = %d\n", x, y, results[0])
|
|||
|
```
|
|||
|
|
|||
|
WebAssembly does not define an encoding strategy for host defined parameters or
|
|||
|
results. This means the encoding rules above are defined by wazero instead. To
|
|||
|
address this, we clarified mapping both in `api.ValueType` and added helper
|
|||
|
functions like `api.EncodeF64`. This allows users conversions typical in Go
|
|||
|
programming, and utilities to avoid ambiguity and edge cases around casting.
|
|||
|
|
|||
|
Alternatively, we could have defined a byte buffer based approach and a binary
|
|||
|
encoding of value types in and out. For example, an empty byte slice would mean
|
|||
|
no values, while a non-empty could use a binary encoding for supported values.
|
|||
|
This could work, but it is more difficult for the normal case of i32 and i64.
|
|||
|
It also shares a struggle with the current approach, which is that value types
|
|||
|
were added after WebAssembly 1.0 and not all of them have an encoding. More on
|
|||
|
this below.
|
|||
|
|
|||
|
In summary, wazero chose an approach for signature mapping because there was
|
|||
|
none, and the one we chose biases towards simplicity with integers and handles
|
|||
|
the rest with documentation and utilities.
|
|||
|
|
|||
|
#### Post 1.0 value types
|
|||
|
|
|||
|
Value types added after WebAssembly 1.0 stressed the current model, as some
|
|||
|
have no encoding or are larger than 64 bits. While problematic, these value
|
|||
|
types are not commonly used in exported (extern) functions. However, some
|
|||
|
decisions were made and detailed below.
|
|||
|
|
|||
|
For example `externref` has no guest representation. wazero chose to map
|
|||
|
references to uint64 as that's the largest value needed to encode a pointer on
|
|||
|
supported platforms. While there are two reference types, `externref` and
|
|||
|
`functype`, the latter is an internal detail of function tables, and the former
|
|||
|
is rarely if ever used in function signatures as of the end of 2022.
|
|||
|
|
|||
|
The only value larger than 64 bits is used for SIMD (`v128`). Vectorizing via
|
|||
|
host functions is not used as of the end of 2022. Even if it were, it would be
|
|||
|
inefficient vs guest vectorization due to host function overhead. In other
|
|||
|
words, the `v128` value type is unlikely to be in an exported function
|
|||
|
signature. That it requires two uint64 values to encode is an internal detail
|
|||
|
and not worth changing the exported function interface `api.Function`, as doing
|
|||
|
so would break all users.
|
|||
|
|
|||
|
### Interfaces, not structs
|
|||
|
|
|||
|
All exported types in public packages, regardless of configuration vs runtime, are interfaces. The primary benefits are
|
|||
|
internal flexibility and avoiding people accidentally mis-initializing by instantiating the types on their own vs using
|
|||
|
the `NewXxx` constructor functions. In other words, there's less support load when things can't be done incorrectly.
|
|||
|
|
|||
|
Here's an example:
|
|||
|
```go
|
|||
|
rt := &RuntimeConfig{} // not initialized properly (fields are nil which shouldn't be)
|
|||
|
rt := RuntimeConfig{} // not initialized properly (should be a pointer)
|
|||
|
rt := wazero.NewRuntimeConfig() // initialized properly
|
|||
|
```
|
|||
|
|
|||
|
There are a few drawbacks to this, notably some work for maintainers.
|
|||
|
* Interfaces are decoupled from the structs implementing them, which means the signature has to be repeated twice.
|
|||
|
* Interfaces have to be documented and guarded at time of use, that 3rd party implementations aren't supported.
|
|||
|
* As of Golang 1.21, interfaces are still [not well supported](https://github.com/golang/go/issues/5860) in godoc.
|
|||
|
|
|||
|
## Config
|
|||
|
|
|||
|
wazero configures scopes such as Runtime and Module using `XxxConfig` types. For example, `RuntimeConfig` configures
|
|||
|
`Runtime` and `ModuleConfig` configure `Module` (instantiation). In all cases, config types begin defaults and can be
|
|||
|
customized by a user, e.g., selecting features or a module name override.
|
|||
|
|
|||
|
### Why don't we make each configuration setting return an error?
|
|||
|
No config types create resources that would need to be closed, nor do they return errors on use. This helps reduce
|
|||
|
resource leaks, and makes chaining easier. It makes it possible to parse configuration (ex by parsing yaml) independent
|
|||
|
of validating it.
|
|||
|
|
|||
|
Instead of:
|
|||
|
```
|
|||
|
cfg, err = cfg.WithFS(fs)
|
|||
|
if err != nil {
|
|||
|
return err
|
|||
|
}
|
|||
|
cfg, err = cfg.WithName(name)
|
|||
|
if err != nil {
|
|||
|
return err
|
|||
|
}
|
|||
|
mod, err = rt.InstantiateModuleWithConfig(ctx, code, cfg)
|
|||
|
if err != nil {
|
|||
|
return err
|
|||
|
}
|
|||
|
```
|
|||
|
|
|||
|
There's only one call site to handle errors:
|
|||
|
```
|
|||
|
cfg = cfg.WithFS(fs).WithName(name)
|
|||
|
mod, err = rt.InstantiateModuleWithConfig(ctx, code, cfg)
|
|||
|
if err != nil {
|
|||
|
return err
|
|||
|
}
|
|||
|
```
|
|||
|
|
|||
|
This allows users one place to look for errors, and also the benefit that if anything internally opens a resource, but
|
|||
|
errs, there's nothing they need to close. In other words, users don't need to track which resources need closing on
|
|||
|
partial error, as that is handled internally by the only code that can read configuration fields.
|
|||
|
|
|||
|
### Why are configuration immutable?
|
|||
|
While it seems certain scopes like `Runtime` won't repeat within a process, they do, possibly in different goroutines.
|
|||
|
For example, some users create a new runtime for each module, and some re-use the same base module configuration with
|
|||
|
only small updates (ex the name) for each instantiation. Making configuration immutable allows them to be safely used in
|
|||
|
any goroutine.
|
|||
|
|
|||
|
Since config are immutable, changes apply via return val, similar to `append` in a slice.
|
|||
|
|
|||
|
For example, both of these are the same sort of error:
|
|||
|
```go
|
|||
|
append(slice, element) // bug as only the return value has the updated slice.
|
|||
|
cfg.WithName(next) // bug as only the return value has the updated name.
|
|||
|
```
|
|||
|
|
|||
|
Here's an example of correct use: re-assigning explicitly or via chaining.
|
|||
|
```go
|
|||
|
cfg = cfg.WithName(name) // explicit
|
|||
|
|
|||
|
mod, err = rt.InstantiateModuleWithConfig(ctx, code, cfg.WithName(name)) // implicit
|
|||
|
if err != nil {
|
|||
|
return err
|
|||
|
}
|
|||
|
```
|
|||
|
|
|||
|
### Why aren't configuration assigned with option types?
|
|||
|
The option pattern is a familiar one in Go. For example, someone defines a type `func (x X) err` and uses it to update
|
|||
|
the target. For example, you could imagine wazero could choose to make `ModuleConfig` from options vs chaining fields.
|
|||
|
|
|||
|
Ex instead of:
|
|||
|
```go
|
|||
|
type ModuleConfig interface {
|
|||
|
WithName(string) ModuleConfig
|
|||
|
WithFS(fs.FS) ModuleConfig
|
|||
|
}
|
|||
|
|
|||
|
struct moduleConfig {
|
|||
|
name string
|
|||
|
fs fs.FS
|
|||
|
}
|
|||
|
|
|||
|
func (c *moduleConfig) WithName(name string) ModuleConfig {
|
|||
|
ret := *c // copy
|
|||
|
ret.name = name
|
|||
|
return &ret
|
|||
|
}
|
|||
|
|
|||
|
func (c *moduleConfig) WithFS(fs fs.FS) ModuleConfig {
|
|||
|
ret := *c // copy
|
|||
|
ret.setFS("/", fs)
|
|||
|
return &ret
|
|||
|
}
|
|||
|
|
|||
|
config := r.NewModuleConfig().WithFS(fs)
|
|||
|
configDerived := config.WithName("name")
|
|||
|
```
|
|||
|
|
|||
|
An option function could be defined, then refactor each config method into an name prefixed option function:
|
|||
|
```go
|
|||
|
type ModuleConfig interface {
|
|||
|
}
|
|||
|
struct moduleConfig {
|
|||
|
name string
|
|||
|
fs fs.FS
|
|||
|
}
|
|||
|
|
|||
|
type ModuleConfigOption func(c *moduleConfig)
|
|||
|
|
|||
|
func ModuleConfigName(name string) ModuleConfigOption {
|
|||
|
return func(c *moduleConfig) {
|
|||
|
c.name = name
|
|||
|
}
|
|||
|
}
|
|||
|
|
|||
|
func ModuleConfigFS(fs fs.FS) ModuleConfigOption {
|
|||
|
return func(c *moduleConfig) {
|
|||
|
c.fs = fs
|
|||
|
}
|
|||
|
}
|
|||
|
|
|||
|
func (r *runtime) NewModuleConfig(opts ...ModuleConfigOption) ModuleConfig {
|
|||
|
ret := newModuleConfig() // defaults
|
|||
|
for _, opt := range opts {
|
|||
|
opt(&ret.config)
|
|||
|
}
|
|||
|
return ret
|
|||
|
}
|
|||
|
|
|||
|
func (c *moduleConfig) WithOptions(opts ...ModuleConfigOption) ModuleConfig {
|
|||
|
ret := *c // copy base config
|
|||
|
for _, opt := range opts {
|
|||
|
opt(&ret.config)
|
|||
|
}
|
|||
|
return ret
|
|||
|
}
|
|||
|
|
|||
|
config := r.NewModuleConfig(ModuleConfigFS(fs))
|
|||
|
configDerived := config.WithOptions(ModuleConfigName("name"))
|
|||
|
```
|
|||
|
|
|||
|
wazero took the path of the former design primarily due to:
|
|||
|
* interfaces provide natural namespaces for their methods, which is more direct than functions with name prefixes.
|
|||
|
* parsing config into function callbacks is more direct vs parsing config into a slice of functions to do the same.
|
|||
|
* in either case derived config is needed and the options pattern is more awkward to achieve that.
|
|||
|
|
|||
|
There are other reasons such as test and debug being simpler without options: the above list is constrained to conserve
|
|||
|
space. It is accepted that the options pattern is common in Go, which is the main reason for documenting this decision.
|
|||
|
|
|||
|
### Why aren't config types deeply structured?
|
|||
|
wazero's configuration types cover the two main scopes of WebAssembly use:
|
|||
|
* `RuntimeConfig`: This is the broadest scope, so applies also to compilation
|
|||
|
and instantiation. e.g. This controls the WebAssembly Specification Version.
|
|||
|
* `ModuleConfig`: This affects modules instantiated after compilation and what
|
|||
|
resources are allowed. e.g. This defines how or if STDOUT is captured. This
|
|||
|
also allows sub-configuration of `FSConfig`.
|
|||
|
|
|||
|
These default to a flat definition each, with lazy sub-configuration only after
|
|||
|
proven to be necessary. A flat structure is easier to work with and is also
|
|||
|
easy to discover. Unlike the option pattern described earlier, more
|
|||
|
configuration in the interface doesn't taint the package namespace, only
|
|||
|
`ModuleConfig`.
|
|||
|
|
|||
|
We default to a flat structure to encourage simplicity. If we eagerly broke out
|
|||
|
all possible configurations into sub-types (e.g. ClockConfig), it would be hard
|
|||
|
to notice configuration sprawl. By keeping the config flat, it is easy to see
|
|||
|
the cognitive load we may be adding to our users.
|
|||
|
|
|||
|
In other words, discomfort adding more configuration is a feature, not a bug.
|
|||
|
We should only add new configuration rarely, and before doing so, ensure it
|
|||
|
will be used. In fact, this is why we support using context fields for
|
|||
|
experimental configuration. By letting users practice, we can find out if a
|
|||
|
configuration was a good idea or not before committing to it, and potentially
|
|||
|
sprawling our types.
|
|||
|
|
|||
|
In reflection, this approach worked well for the nearly 1.5 year period leading
|
|||
|
to version 1.0. We've only had to create a single sub-configuration, `FSConfig`,
|
|||
|
and it was well understood why when it occurred.
|
|||
|
|
|||
|
## Why does `ModuleConfig.WithStartFunctions` default to `_start`?
|
|||
|
|
|||
|
We formerly had functions like `StartWASICommand` that would verify
|
|||
|
preconditions and start WASI's `_start` command. However, this caused confusion
|
|||
|
because both many languages compiled a WASI dependency, and many did so
|
|||
|
inconsistently.
|
|||
|
|
|||
|
The conflict is that exported functions need to use features the language
|
|||
|
runtime provides, such as garbage collection. There's a "chicken-egg problem"
|
|||
|
where `_start` needs to complete in order for exported behavior to work.
|
|||
|
|
|||
|
For example, unlike `GOOS=wasip1` in Go 1.21, TinyGo's "wasi" target supports
|
|||
|
function exports. So, the only way to use FFI style is via the "wasi" target.
|
|||
|
Not explicitly calling `_start` before an ABI such as wapc-go, would crash, due
|
|||
|
to setup not happening (e.g. to implement `panic`). Other embedders such as
|
|||
|
Envoy also called `_start` for the same reason. To avoid a common problem for
|
|||
|
users unaware of WASI, and also to simplify normal use of WASI (e.g. `main`),
|
|||
|
we added `_start` to `ModuleConfig.WithStartFunctions`.
|
|||
|
|
|||
|
In cases of multiple initializers, such as in wapc-go, users can override this
|
|||
|
to add the others *after* `_start`. Users who want to explicitly control
|
|||
|
`_start`, such as some of our unit tests, can clear the start functions and
|
|||
|
remove it.
|
|||
|
|
|||
|
This decision was made in 2022, and holds true in 2023, even with the
|
|||
|
introduction of "wasix". It holds because "wasix" is backwards compatible with
|
|||
|
"wasip1". In the future, there will be other ways to start applications, and
|
|||
|
may not be backwards compatible with "wasip1".
|
|||
|
|
|||
|
Most notably WASI "Preview 2" is not implemented in a way compatible with
|
|||
|
wasip1. Its start function is likely to be different, and defined in the
|
|||
|
wasi-cli "world". When the design settles, and it is implemented by compilers,
|
|||
|
wazero will attempt to support "wasip2". However, it won't do so in a way that
|
|||
|
breaks existing compilers.
|
|||
|
|
|||
|
In other words, we won't remove `_start` if "wasip2" continues a path of an
|
|||
|
alternate function name. If we did, we'd break existing users despite our
|
|||
|
compatibility promise saying we don't. The most likely case is that when we
|
|||
|
build-in something incompatible with "wasip1", that start function will be
|
|||
|
added to the start functions list in addition to `_start`.
|
|||
|
|
|||
|
See http://wasix.org
|
|||
|
See https://github.com/WebAssembly/wasi-cli
|
|||
|
|
|||
|
## Runtime == Engine+Store
|
|||
|
wazero defines a single user-type which combines the specification concept of `Store` with the unspecified `Engine`
|
|||
|
which manages them.
|
|||
|
|
|||
|
### Why not multi-store?
|
|||
|
Multi-store isn't supported as the extra tier complicates lifecycle and locking. Moreover, in practice it is unusual for
|
|||
|
there to be an engine that has multiple stores which have multiple modules. More often, it is the case that there is
|
|||
|
either 1 engine with 1 store and multiple modules, or 1 engine with many stores, each having 1 non-host module. In worst
|
|||
|
case, a user can use multiple runtimes until "multi-store" is better understood.
|
|||
|
|
|||
|
If later, we have demand for multiple stores, that can be accomplished by overload. e.g. `Runtime.InstantiateInStore` or
|
|||
|
`Runtime.Store(name) Store`.
|
|||
|
|
|||
|
## Exit
|
|||
|
|
|||
|
### Why do we only return a `sys.ExitError` on a non-zero exit code?
|
|||
|
|
|||
|
It is reasonable to think an exit error should be returned, even if the code is
|
|||
|
success (zero). Even on success, the module is no longer functional. For
|
|||
|
example, function exports would error later. However, wazero does not. The only
|
|||
|
time `sys.ExitError` is on error (non-zero).
|
|||
|
|
|||
|
This decision was to improve performance and ergonomics for guests that both
|
|||
|
use WASI (have a `_start` function), and also allow custom exports.
|
|||
|
Specifically, Rust, TinyGo and normal wasi-libc, don't exit the module during
|
|||
|
`_start`. If they did, it would invalidate their function exports. This means
|
|||
|
it is unlikely most compilers will change this behavior.
|
|||
|
|
|||
|
`GOOS=waspi1` from Go 1.21 does exit during `_start`. However, it doesn't
|
|||
|
support other exports besides `_start`, and `_start` is not defined to be
|
|||
|
called multiple times anyway.
|
|||
|
|
|||
|
Since `sys.ExitError` is not always returned, we added `Module.IsClosed` for
|
|||
|
defensive checks. This helps integrators avoid calling functions which will
|
|||
|
always fail.
|
|||
|
|
|||
|
### Why panic with `sys.ExitError` after a host function exits?
|
|||
|
|
|||
|
Currently, the only portable way to stop processing code is via panic. For
|
|||
|
example, WebAssembly "trap" instructions, such as divide by zero, are
|
|||
|
implemented via panic. This ensures code isn't executed after it.
|
|||
|
|
|||
|
When code reaches the WASI `proc_exit` instruction, we need to stop processing.
|
|||
|
Regardless of the exit code, any code invoked after exit would be in an
|
|||
|
inconsistent state. This is likely why unreachable instructions are sometimes
|
|||
|
inserted after exit: https://github.com/emscripten-core/emscripten/issues/12322
|
|||
|
|
|||
|
## WASI
|
|||
|
|
|||
|
Unfortunately, (WASI Snapshot Preview 1)[https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md] is not formally defined enough, and has APIs with ambiguous semantics.
|
|||
|
This section describes how Wazero interprets and implements the semantics of several WASI APIs that may be interpreted differently by different wasm runtimes.
|
|||
|
Those APIs may affect the portability of a WASI application.
|
|||
|
|
|||
|
### Why don't we attempt to pass wasi-testsuite on user-defined `fs.FS`?
|
|||
|
|
|||
|
While most cases work fine on an `os.File` based implementation, we won't
|
|||
|
promise wasi-testsuite compatibility on user defined wrappers of `os.DirFS`.
|
|||
|
The only option for real systems is to use our `sysfs.FS`.
|
|||
|
|
|||
|
There are a lot of areas where windows behaves differently, despite the
|
|||
|
`os.File` abstraction. This goes well beyond file locking concerns (e.g.
|
|||
|
`EBUSY` errors on open files). For example, errors like `ACCESS_DENIED` aren't
|
|||
|
properly mapped to `EPERM`. There are trickier parts too. `FileInfo.Sys()`
|
|||
|
doesn't return enough information to build inodes needed for WASI. To rebuild
|
|||
|
them requires the full path to the underlying file, not just its directory
|
|||
|
name, and there's no way for us to get that information. At one point we tried,
|
|||
|
but in practice things became tangled and functionality such as read-only
|
|||
|
wrappers became untenable. Finally, there are version-specific behaviors which
|
|||
|
are difficult to maintain even in our own code. For example, go 1.20 opens
|
|||
|
files in a different way than versions before it.
|
|||
|
|
|||
|
### Why aren't WASI rules enforced?
|
|||
|
|
|||
|
The [snapshot-01](https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md) version of WASI has a
|
|||
|
number of rules for a "command module", but only the memory export rule is enforced. If a "_start" function exists, it
|
|||
|
is enforced to be the correct signature and succeed, but the export itself isn't enforced. It follows that this means
|
|||
|
exports are not required to be contained to a "_start" function invocation. Finally, the "__indirect_function_table"
|
|||
|
export is also not enforced.
|
|||
|
|
|||
|
The reason for the exceptions are that implementations aren't following the rules. For example, TinyGo doesn't export
|
|||
|
"__indirect_function_table", so crashing on this would make wazero unable to run TinyGo modules. Similarly, modules
|
|||
|
loaded by wapc-go don't always define a "_start" function. Since "snapshot-01" is not a proper version, and certainly
|
|||
|
not a W3C recommendation, there's no sense in breaking users over matters like this.
|
|||
|
|
|||
|
### Why is I/O configuration not coupled to WASI?
|
|||
|
|
|||
|
WebAssembly System Interfaces (WASI) is a formalization of a practice that can be done anyway: Define a host function to
|
|||
|
access a system interface, such as writing to STDOUT. WASI stalled at snapshot-01 and as of early 2023, is being
|
|||
|
rewritten entirely.
|
|||
|
|
|||
|
This instability implies a need to transition between WASI specs, which places wazero in a position that requires
|
|||
|
decoupling. For example, if code uses two different functions to call `fd_write`, the underlying configuration must be
|
|||
|
centralized and decoupled. Otherwise, calls using the same file descriptor number will end up writing to different
|
|||
|
places.
|
|||
|
|
|||
|
In short, wazero defined system configuration in `ModuleConfig`, not a WASI type. This allows end-users to switch from
|
|||
|
one spec to another with minimal impact. This has other helpful benefits, as centralized resources are simpler to close
|
|||
|
coherently (ex via `Module.Close`).
|
|||
|
|
|||
|
In reflection, this worked well as more ABI became usable in wazero.
|
|||
|
|
|||
|
### Background on `ModuleConfig` design
|
|||
|
|
|||
|
WebAssembly 1.0 (20191205) specifies some aspects to control isolation between modules ([sandboxing](https://en.wikipedia.org/wiki/Sandbox_(computer_security))).
|
|||
|
For example, `wasm.Memory` has size constraints and each instance of it is isolated from each other. While `wasm.Memory`
|
|||
|
can be shared, by exporting it, it is not exported by default. In fact a WebAssembly Module (Wasm) has no memory by
|
|||
|
default.
|
|||
|
|
|||
|
While memory is defined in WebAssembly 1.0 (20191205), many aspects are not. Let's use an example of `exec.Cmd` as for
|
|||
|
example, a WebAssembly System Interfaces (WASI) command is implemented as a module with a `_start` function, and in many
|
|||
|
ways acts similar to a process with a `main` function.
|
|||
|
|
|||
|
To capture "hello world" written to the console (stdout a.k.a. file descriptor 1) in `exec.Cmd`, you would set the
|
|||
|
`Stdout` field accordingly, perhaps to a buffer. In WebAssembly 1.0 (20191205), the only way to perform something like
|
|||
|
this is via a host function (ex `HostModuleFunctionBuilder`) and internally copy memory corresponding to that string
|
|||
|
to a buffer.
|
|||
|
|
|||
|
WASI implements system interfaces with host functions. Concretely, to write to console, a WASI command `Module` imports
|
|||
|
"fd_write" from "wasi_snapshot_preview1" and calls it with the `fd` parameter set to 1 (STDOUT).
|
|||
|
|
|||
|
The [snapshot-01](https://github.com/WebAssembly/WASI/blob/snapshot-01/phases/snapshot/docs.md) version of WASI has no
|
|||
|
means to declare configuration, although its function definitions imply configuration for example if fd 1 should exist,
|
|||
|
and if so where should it write. Moreover, snapshot-01 was last updated in late 2020 and the specification is being
|
|||
|
completely rewritten as of early 2022. This means WASI as defined by "snapshot-01" will not clarify aspects like which
|
|||
|
file descriptors are required. While it is possible a subsequent version may, it is too early to tell as no version of
|
|||
|
WASI has reached a stage near W3C recommendation. Even if it did, module authors are not required to only use WASI to
|
|||
|
write to console, as they can define their own host functions, such as they did before WASI existed.
|
|||
|
|
|||
|
wazero aims to serve Go developers as a primary function, and help them transition between WASI specifications. In
|
|||
|
order to do this, we have to allow top-level configuration. To ensure isolation by default, `ModuleConfig` has WithXXX
|
|||
|
that override defaults to no-op or empty. One `ModuleConfig` instance is used regardless of how many times the same WASI
|
|||
|
functions are imported. The nil defaults allow safe concurrency in these situations, as well lower the cost when they
|
|||
|
are never used. Finally, a one-to-one mapping with `Module` allows the module to close the `ModuleConfig` instead of
|
|||
|
confusing users with another API to close.
|
|||
|
|
|||
|
Naming, defaults and validation rules of aspects like `STDIN` and `Environ` are intentionally similar to other Go
|
|||
|
libraries such as `exec.Cmd` or `syscall.SetEnv`, and differences called out where helpful. For example, there's no goal
|
|||
|
to emulate any operating system primitive specific to Windows (such as a 'c:\' drive). Moreover, certain defaults
|
|||
|
working with real system calls are neither relevant nor safe to inherit: For example, `exec.Cmd` defaults to read STDIN
|
|||
|
from a real file descriptor ("/dev/null"). Defaulting to this, vs reading `io.EOF`, would be unsafe as it can exhaust
|
|||
|
file descriptors if resources aren't managed properly. In other words, blind copying of defaults isn't wise as it can
|
|||
|
violate isolation or endanger the embedding process. In summary, we try to be similar to normal Go code, but often need
|
|||
|
act differently and document `ModuleConfig` is more about emulating, not necessarily performing real system calls.
|
|||
|
|
|||
|
## File systems
|
|||
|
|
|||
|
### Motivation on `sys.FS`
|
|||
|
|
|||
|
The `sys.FS` abstraction in wazero was created because of limitations in
|
|||
|
`fs.FS`, and `fs.File` in Go. Compilers targeting `wasip1` may access
|
|||
|
functionality that writes new files. The ability to overcome this was requested
|
|||
|
even before wazero was named this, via issue #21 in March 2021.
|
|||
|
|
|||
|
A month later, golang/go#45757 was raised by someone else on the same topic. As
|
|||
|
of July 2023, this has not resolved to a writeable file system abstraction.
|
|||
|
|
|||
|
Over the next year more use cases accumulated, consolidated in March 2022 into
|
|||
|
#390. This closed in January 2023 with a milestone of providing more
|
|||
|
functionality, limited to users giving a real directory. This didn't yet expose
|
|||
|
a file abstraction for general purpose use. Internally, this used `os.File`.
|
|||
|
However, a wasm module instance is a virtual machine. Only supporting `os.File`
|
|||
|
breaks sand-boxing use cases. Moreover, `os.File` is not an interface. Even
|
|||
|
though this abstracts functionality, it does allow interception use cases.
|
|||
|
|
|||
|
Hence, a few days later in January 2023, we had more issues asking to expose an
|
|||
|
abstraction, #1013 and later #1532, on use cases like masking access to files.
|
|||
|
In other words, the use case requests never stopped, and aren't solved by
|
|||
|
exposing only real files.
|
|||
|
|
|||
|
In summary, the primary motivation for exposing a replacement for `fs.FS` and
|
|||
|
`fs.File` was around repetitive use case requests for years, around
|
|||
|
interception and the ability to create new files, both virtual and real files.
|
|||
|
While some use cases are solved with real files, not all are. Regardless, an
|
|||
|
interface approach is necessary to ensure users can intercept I/O operations.
|
|||
|
|
|||
|
### Why doesn't `sys.File` have a `Fd()` method?
|
|||
|
|
|||
|
There are many features we could expose. We could make File expose underlying
|
|||
|
file descriptors in case they are supported, for integration of system calls
|
|||
|
that accept multiple ones, namely `poll` for multiplexing. This special case is
|
|||
|
described in a subsequent section.
|
|||
|
|
|||
|
As noted above, users have been asking for a file abstraction for over two
|
|||
|
years, and a common answer was to wait. Making users wait is a problem,
|
|||
|
especially so long. Good reasons to make people wait are stabilization. Edge
|
|||
|
case features are not a great reason to hold abstractions from users.
|
|||
|
|
|||
|
Another reason is implementation difficulty. Go did not attempt to abstract
|
|||
|
file descriptors. For example, unlike `fs.ReadFile` there is no `fs.FdFile`
|
|||
|
interface. Most likely, this is because file descriptors are an implementation
|
|||
|
detail of common features. Programming languages, including Go, do not require
|
|||
|
end users to know about file descriptors. Types such as `fs.File` can be used
|
|||
|
without any knowledge of them. Implementations may or may not have file
|
|||
|
descriptors. For example, in Go, `os.DirFS` has underlying file descriptors
|
|||
|
while `embed.FS` does not.
|
|||
|
|
|||
|
Despite this, some may want to expose a non-standard interface because
|
|||
|
`os.File` has `Fd() uintptr` to return a file descriptor. Mainly, this is
|
|||
|
handy to integrate with `syscall` package functions (on `GOOS` values that
|
|||
|
declare them). Notice, though that `uintptr` is unsafe and not an abstraction.
|
|||
|
Close inspection will find some `os.File` types internally use `poll.FD`
|
|||
|
instead, yet this is not possible to use abstractly because that type is not
|
|||
|
exposed. For example, `plan9` uses a different type than `poll.FD`. In other
|
|||
|
words, even in real files, `Fd()` is not wholly portable, despite it being
|
|||
|
useful on many operating systems with the `syscall` package.
|
|||
|
|
|||
|
The reasons above, why Go doesn't abstract `FdFile` interface are a subset of
|
|||
|
reasons why `sys.File` does not. If we exposed `File.Fd()` we not only would
|
|||
|
have to declare all the edge cases that Go describes including impact of
|
|||
|
finalizers, we would have to describe these in terms of virtualized files.
|
|||
|
Then, we would have to reason with this value vs our existing virtualized
|
|||
|
`sys.FileTable`, mapping whatever type we return to keys in that table, also
|
|||
|
in consideration of garbage collection impact. The combination of issues like
|
|||
|
this could lead down a path of not implementing a file system abstraction at
|
|||
|
all, and instead a weak key mapped abstraction of the `syscall` package. Once
|
|||
|
we finished with all the edge cases, we would have lost context of the original
|
|||
|
reason why we started.. simply to allow file write access!
|
|||
|
|
|||
|
When wazero attempts to do more than what the Go programming language team, it
|
|||
|
has to be carefully evaluated, to:
|
|||
|
* Be possible to implement at least for `os.File` backed files
|
|||
|
* Not be confusing or cognitively hard for virtual file systems and normal use.
|
|||
|
* Affordable: custom code is solely the responsible by the core team, a much
|
|||
|
smaller group of individuals than who maintain the Go programming language.
|
|||
|
|
|||
|
Due to problems well known in Go, consideration of the end users who constantly
|
|||
|
ask for basic file system functionality, and the difficulty virtualizing file
|
|||
|
descriptors at multiple levels, we don't expose `Fd()` and likely won't ever
|
|||
|
expose `Fd()` on `sys.File`.
|
|||
|
|
|||
|
### Why does `sys.File` have a `Poll()` method, while `sys.FS` does not?
|
|||
|
|
|||
|
wazero exposes `File.Poll` which allows one-at-a-time poll use cases,
|
|||
|
requested by multiple users. This not only includes abstract tests such as
|
|||
|
Go 1.21 `GOOS=wasip1`, but real use cases including python and container2wasm
|
|||
|
repls, as well listen sockets. The main use cases is non-blocking poll on a
|
|||
|
single file. Being a single file, this has no risk of problems such as
|
|||
|
head-of-line blocking, even when emulated.
|
|||
|
|
|||
|
The main use case of multi-poll are bidirectional network services, something
|
|||
|
not used in `GOOS=wasip1` standard libraries, but could be in the future.
|
|||
|
Moving forward without a multi-poller allows wazero to expose its file system
|
|||
|
abstraction instead of continuing to hold back it back for edge cases. We'll
|
|||
|
continue discussion below regardless, as rationale was requested.
|
|||
|
|
|||
|
You can loop through multiple `sys.File`, using `File.Poll` to see if an event
|
|||
|
is ready, but there is a head-of-line blocking problem. If a long timeout is
|
|||
|
used, bad luck could have a file that has nothing to read or write before one
|
|||
|
that does. This could cause more blocking than necessary, even if you could
|
|||
|
poll the others just after with a zero timeout. What's worse than this is if
|
|||
|
unlimited blocking was used (`timeout=-1`). The host implementations could use
|
|||
|
goroutines to avoid this, but interrupting a "forever" poll is problematic. All
|
|||
|
of these are reasons to consider a multi-poll API, but do not require exporting
|
|||
|
`File.Fd()`.
|
|||
|
|
|||
|
Should multi-poll becomes critical, `sys.FS` could expose a `Poll` function
|
|||
|
like below, despite it being the non-portable, complicated if possible to
|
|||
|
implement on all platforms and virtual file systems.
|
|||
|
```go
|
|||
|
ready, errno := fs.Poll([]sys.PollFile{{f1, sys.POLLIN}, {f2, sys.POLLOUT}}, timeoutMillis)
|
|||
|
```
|
|||
|
|
|||
|
A real filesystem could handle this by using an approach like the internal
|
|||
|
`unix.Poll` function in Go, passing file descriptors on unix platforms, or
|
|||
|
returning `sys.ENOSYS` for unsupported operating systems. Implementation for
|
|||
|
virtual files could have a strategy around timeout to avoid the worst case of
|
|||
|
head-of-line blocking (unlimited timeout).
|
|||
|
|
|||
|
Let's remember that when designing abstractions, it is not best to add an
|
|||
|
interface for everything. Certainly, Go doesn't, as evidenced by them not
|
|||
|
exposing `poll.FD` in `os.File`! Such a multi-poll could be limited to
|
|||
|
built-in filesystems in the wazero repository, avoiding complexity of trying to
|
|||
|
support and test this abstractly. This would still permit multiplexing for CLI
|
|||
|
users, and also permit single file polling as exists now.
|
|||
|
|
|||
|
### Why doesn't wazero implement the working directory?
|
|||
|
|
|||
|
An early design of wazero's API included a `WithWorkDirFS` which allowed
|
|||
|
control over which file a relative path such as "./config.yml" resolved to,
|
|||
|
independent of the root file system. This intended to help separate concerns
|
|||
|
like mutability of files, but it didn't work and was removed.
|
|||
|
|
|||
|
Compilers that target wasm act differently with regard to the working
|
|||
|
directory. For example, wasi-libc, used by TinyGo,
|
|||
|
tracks working directory changes in compiled wasm instead: initially "/" until
|
|||
|
code calls `chdir`. Zig assumes the first pre-opened file descriptor is the
|
|||
|
working directory.
|
|||
|
|
|||
|
The only place wazero can standardize a layered concern is via a host function.
|
|||
|
Since WASI doesn't use host functions to track the working directory, we can't
|
|||
|
standardize the storage and initial value of it.
|
|||
|
|
|||
|
Meanwhile, code may be able to affect the working directory by compiling
|
|||
|
`chdir` into their main function, using an argument or ENV for the initial
|
|||
|
value (possibly `PWD`). Those unable to control the compiled code should only
|
|||
|
use absolute paths in configuration.
|
|||
|
|
|||
|
See
|
|||
|
* https://github.com/golang/go/blob/go1.20/src/syscall/fs_js.go#L324
|
|||
|
* https://github.com/WebAssembly/wasi-libc/pull/214#issue-673090117
|
|||
|
* https://github.com/ziglang/zig/blob/53a9ee699a35a3d245ab6d1dac1f0687a4dcb42c/src/main.zig#L32
|
|||
|
|
|||
|
### Why ignore the error returned by io.Reader when n > 1?
|
|||
|
|
|||
|
Per https://pkg.go.dev/io#Reader, if we receive an error, any bytes read should
|
|||
|
be processed first. At the syscall abstraction (`fd_read`), the caller is the
|
|||
|
processor, so we can't process the bytes inline and also return the error (as
|
|||
|
`EIO`).
|
|||
|
|
|||
|
Let's assume we want to return the bytes read on error to the caller. This
|
|||
|
implies we at least temporarily ignore the error alongside them. The choice
|
|||
|
remaining is whether to persist the error returned with the read until a
|
|||
|
possible next call, or ignore the error.
|
|||
|
|
|||
|
If we persist an error returned, it would be coupled to a file descriptor, but
|
|||
|
effectively it is boolean as this case coerces to `EIO`. If we track a "last
|
|||
|
error" on a file descriptor, it could be complicated for a couple reasons
|
|||
|
including whether the error is transient or permanent, or if the error would
|
|||
|
apply to any FD operation, or just read. Finally, there may never be a
|
|||
|
subsequent read as perhaps the bytes leading up to the error are enough to
|
|||
|
satisfy the processor.
|
|||
|
|
|||
|
This decision boils down to whether or not to track an error bit per file
|
|||
|
descriptor or not. If not, the assumption is that a subsequent operation would
|
|||
|
also error, this time without reading any bytes.
|
|||
|
|
|||
|
The current opinion is to go with the simplest path, which is to return the
|
|||
|
bytes read and ignore the error the there were any. Assume a subsequent
|
|||
|
operation will err if it needs to. This helps reduce the complexity of the code
|
|||
|
in wazero and also accommodates the scenario where the bytes read are enough to
|
|||
|
satisfy its processor.
|
|||
|
|
|||
|
### File descriptor allocation strategy
|
|||
|
|
|||
|
File descriptor allocation currently uses a strategy similar the one implemented
|
|||
|
by unix systems: when opening a file, the lowest unused number is picked.
|
|||
|
|
|||
|
The WASI standard documents that programs cannot expect that file descriptor
|
|||
|
numbers will be allocated with a lowest-first strategy, and they should instead
|
|||
|
assume the values will be random. Since _random_ is a very imprecise concept in
|
|||
|
computers, we technically satisfying the implementation with the descriptor
|
|||
|
allocation strategy we use in Wazero. We could imagine adding more _randomness_
|
|||
|
to the descriptor selection process, however this should never be used as a
|
|||
|
security measure to prevent applications from guessing the next file number so
|
|||
|
there are no strong incentives to complicate the logic.
|
|||
|
|
|||
|
### Why does `FSConfig.WithDirMount` not match behaviour with `os.DirFS`?
|
|||
|
|
|||
|
It may seem that we should require any feature that seems like a standard
|
|||
|
library in Go, to behave the same way as the standard library. Doing so would
|
|||
|
present least surprise to Go developers. In the case of how we handle
|
|||
|
filesystems, we break from that as it is incompatible with the expectations of
|
|||
|
WASI, the most commonly implemented filesystem ABI.
|
|||
|
|
|||
|
The main reason is that `os.DirFS` is a virtual filesystem abstraction while
|
|||
|
WASI is an abstraction over syscalls. For example, the signature of `fs.Open`
|
|||
|
does not permit use of flags. This creates conflict on what default behaviors
|
|||
|
to take when Go implemented `os.DirFS`. On the other hand, `path_open` can pass
|
|||
|
flags, and in fact tests require them to be honored in specific ways.
|
|||
|
|
|||
|
This conflict requires us to choose what to be more compatible with, and which
|
|||
|
type of user to surprise the least. We assume there will be more developers
|
|||
|
compiling code to wasm than developers of custom filesystem plugins, and those
|
|||
|
compiling code to wasm will be better served if we are compatible with WASI.
|
|||
|
Hence on conflict, we prefer WASI behavior vs the behavior of `os.DirFS`.
|
|||
|
|
|||
|
See https://github.com/WebAssembly/wasi-testsuite
|
|||
|
See https://github.com/golang/go/issues/58141
|
|||
|
|
|||
|
## Why is our `Readdir` function more like Go's `os.File` than POSIX `readdir`?
|
|||
|
|
|||
|
At one point we attempted to move from a bulk `Readdir` function to something
|
|||
|
more like the POSIX `DIR` struct, exposing functions like `telldir`, `seekdir`
|
|||
|
and `readdir`. However, we chose the design more like `os.File.Readdir`,
|
|||
|
because it performs and fits wasip1 better.
|
|||
|
|
|||
|
### wasip1/wasix
|
|||
|
|
|||
|
`fd_readdir` in wasip1 (and so also wasix) is like `getdents` in Linux, not
|
|||
|
`readdir` in POSIX. `getdents` is more like Go's `os.File.Readdir`.
|
|||
|
|
|||
|
We currently have an internal type `sys.DirentCache` which only is used by
|
|||
|
wasip1 or wasix. When `HostModuleBuilder` adds support for instantiation state,
|
|||
|
we could move this to the `wasi_snapshot_preview1` package. Meanwhile, all
|
|||
|
filesystem code is internal anyway, so this special-case is acceptable.
|
|||
|
|
|||
|
### wasip2
|
|||
|
|
|||
|
`directory-entry-stream` in wasi-filesystem preview2 is defined in component
|
|||
|
model, not an ABI, but in wasmtime it is a consuming iterator. A consuming
|
|||
|
iterator is easy to support with anything (like `Readdir(1)`), even if it is
|
|||
|
inefficient as you can neither bulk read nor skip. The implementation of the
|
|||
|
preview1 adapter (uses preview2) confirms this. They use a dirent cache similar
|
|||
|
in some ways to our `sysfs.DirentCache`. As there is no seek concept in
|
|||
|
preview2, they interpret the cookie as numeric and read on repeat entries when
|
|||
|
a cache wasn't available. Note: we currently do not skip-read like this as it
|
|||
|
risks buffering large directories, and no user has requested entries before the
|
|||
|
cache, yet.
|
|||
|
|
|||
|
Regardless, wasip2 is not complete until the end of 2023. We can defer design
|
|||
|
discussion until after it is stable and after the reference impl wasmtime
|
|||
|
implements it.
|
|||
|
|
|||
|
See
|
|||
|
* https://github.com/WebAssembly/wasi-filesystem/blob/ef9fc87c07323a6827632edeb6a7388b31266c8e/example-world.md#directory_entry_stream
|
|||
|
* https://github.com/bytecodealliance/wasmtime/blob/b741f7c79d72492d17ab8a29c8ffe4687715938e/crates/wasi/src/preview2/preview2/filesystem.rs#L286-L296
|
|||
|
* https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L2131-L2137
|
|||
|
* https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L936
|
|||
|
|
|||
|
### wasip3
|
|||
|
|
|||
|
`directory-entry-stream` is documented to change significantly in wasip3 moving
|
|||
|
from synchronous to synchronous streams. This is dramatically different than
|
|||
|
POSIX `readdir` which is synchronous.
|
|||
|
|
|||
|
Regardless, wasip3 is not complete until after wasip2, which means 2024 or
|
|||
|
later. We can defer design discussion until after it is stable and after the
|
|||
|
reference impl wasmtime implements it.
|
|||
|
|
|||
|
See
|
|||
|
* https://github.com/WebAssembly/WASI/blob/ddfe3d1dda5d1473f37ecebc552ae20ce5fd319a/docs/WitInWasi.md#Streams
|
|||
|
* https://docs.google.com/presentation/d/1MNVOZ8hdofO3tI0szg_i-Yoy0N2QPU2C--LzVuoGSlE/edit#slide=id.g1270ef7d5b6_0_662
|
|||
|
|
|||
|
### How do we implement `Pread` with an `fs.File`?
|
|||
|
|
|||
|
`ReadAt` is the Go equivalent to `pread`: it does not affect, and is not
|
|||
|
affected by, the underlying file offset. Unfortunately, `io.ReaderAt` is not
|
|||
|
implemented by all `fs.File`. For example, as of Go 1.19, `embed.openFile` does
|
|||
|
not.
|
|||
|
|
|||
|
The initial implementation of `fd_pread` instead used `Seek`. To avoid a
|
|||
|
regression, we fall back to `io.Seeker` when `io.ReaderAt` is not supported.
|
|||
|
|
|||
|
This requires obtaining the initial file offset, seeking to the intended read
|
|||
|
offset, and resetting the file offset the initial state. If this final seek
|
|||
|
fails, the file offset is left in an undefined state. This is not thread-safe.
|
|||
|
|
|||
|
While seeking per read seems expensive, the common case of `embed.openFile` is
|
|||
|
only accessing a single int64 field, which is cheap.
|
|||
|
|
|||
|
### Pre-opened files
|
|||
|
|
|||
|
WASI includes `fd_prestat_get` and `fd_prestat_dir_name` functions used to
|
|||
|
learn any directory paths for file descriptors open at initialization time.
|
|||
|
|
|||
|
For example, `__wasilibc_register_preopened_fd` scans any file descriptors past
|
|||
|
STDERR (1) and invokes `fd_prestat_dir_name` to learn any path prefixes they
|
|||
|
correspond to. Zig's `preopensAlloc` does similar. These pre-open functions are
|
|||
|
not used again after initialization.
|
|||
|
|
|||
|
wazero supports stdio pre-opens followed by any mounts e.g `.:/`. The guest
|
|||
|
path is a directory and its name, e.g. "/" is returned by `fd_prestat_dir_name`
|
|||
|
for file descriptor 3 (STDERR+1). The first longest match wins on multiple
|
|||
|
pre-opens, which allows a path like "/tmp" to match regardless of order vs "/".
|
|||
|
|
|||
|
See
|
|||
|
* https://github.com/WebAssembly/wasi-libc/blob/a02298043ff551ce1157bc2ee7ab74c3bffe7144/libc-bottom-half/sources/preopens.c
|
|||
|
* https://github.com/ziglang/zig/blob/9cb06f3b8bf9ea6b5e5307711bc97328762d6a1d/lib/std/fs/wasi.zig#L50-L53
|
|||
|
|
|||
|
### fd_prestat_dir_name
|
|||
|
|
|||
|
`fd_prestat_dir_name` is a WASI function to return the path of the pre-opened
|
|||
|
directory of a file descriptor. It has the following three parameters, and the
|
|||
|
third `path_len` has ambiguous semantics.
|
|||
|
|
|||
|
* `fd`: a file descriptor
|
|||
|
* `path`: the offset for the result path
|
|||
|
* `path_len`: In wazero, `FdPrestatDirName` writes the result path string to
|
|||
|
`path` offset for the exact length of `path_len`.
|
|||
|
|
|||
|
Wasmer considers `path_len` to be the maximum length instead of the exact
|
|||
|
length that should be written.
|
|||
|
See https://github.com/wasmerio/wasmer/blob/3463c51268ed551933392a4063bd4f8e7498b0f6/lib/wasi/src/syscalls/mod.rs#L764
|
|||
|
|
|||
|
The semantics in wazero follows that of wasmtime.
|
|||
|
See https://github.com/bytecodealliance/wasmtime/blob/2ca01ae9478f199337cf743a6ab543e8c3f3b238/crates/wasi-common/src/snapshots/preview_1.rs#L578-L582
|
|||
|
|
|||
|
Their semantics match when `path_len` == the length of `path`, so in practice
|
|||
|
this difference won't matter match.
|
|||
|
|
|||
|
## fd_readdir
|
|||
|
|
|||
|
### Why does "wasi_snapshot_preview1" require dot entries when POSIX does not?
|
|||
|
|
|||
|
In October 2019, WASI project knew requiring dot entries ("." and "..") was not
|
|||
|
documented in preview1, not required by POSIX and problematic to synthesize.
|
|||
|
For example, Windows runtimes backed by `FindNextFileW` could not return these.
|
|||
|
A year later, the tag representing WASI preview 1 (`snapshot-01`) was made.
|
|||
|
This did not include the requested change of making dot entries optional.
|
|||
|
|
|||
|
The `phases/snapshot/docs.md` document was altered in subsequent years in
|
|||
|
significant ways, often in lock-step with wasmtime or wasi-libc. In January
|
|||
|
2022, `sock_accept` was added to `phases/snapshot/docs.md`, a document later
|
|||
|
renamed to later renamed to `legacy/preview1/docs.md`.
|
|||
|
|
|||
|
As a result, the ABI and behavior remained unstable: The `snapshot-01` tag was
|
|||
|
not an effective basis of portability. A test suite was requested well before
|
|||
|
this tag, in April 2019. Meanwhile, compliance had no meaning. Developers had
|
|||
|
to track changes to the latest doc, while clarifying with wasi-libc or wasmtime
|
|||
|
behavior. This lack of stability could have permitted a fix to the dot entries
|
|||
|
problem, just as it permitted changes desired by other users.
|
|||
|
|
|||
|
In November 2022, the wasi-testsuite project began and started solidifying
|
|||
|
expectations. This quickly led to changes in runtimes and the spec doc. WASI
|
|||
|
began importing tests from wasmtime as required behaviors for all runtimes.
|
|||
|
Some changes implied changes to wasi-libc. For example, `readdir` began to
|
|||
|
imply inode fan-outs, which caused performance regressions. Most notably a
|
|||
|
test merged in January required dot entries. Tests were merged without running
|
|||
|
against any runtime, and even when run ad-hoc only against Linux. Hence,
|
|||
|
portability issues mentioned over three years earlier did not trigger any
|
|||
|
failure until wazero (which tests Windows) noticed.
|
|||
|
|
|||
|
In the same month, wazero requested to revert this change primarily because
|
|||
|
Go does not return them from `os.ReadDir`, and materializing them is
|
|||
|
complicated due to tests also requiring inodes. Moreover, they are discarded by
|
|||
|
not just Go, but other common programming languages. This was rejected by the
|
|||
|
WASI lead for preview1, but considered for the completely different ABI named
|
|||
|
preview2.
|
|||
|
|
|||
|
In February 2023, the WASI chair declared that new rule requiring preview1 to
|
|||
|
return dot entries "was decided by the subgroup as a whole", citing meeting
|
|||
|
notes. According to these notes, the WASI lead stated incorrectly that POSIX
|
|||
|
conformance required returning dot entries, something it explicitly says are
|
|||
|
optional. In other words, he said filtering them out would make Preview1
|
|||
|
non-conforming, and asked if anyone objects to this. The co-chair was noted to
|
|||
|
say "Because there are existing P1 programs, we shouldn’t make changes like
|
|||
|
this." No other were recorded to say anything.
|
|||
|
|
|||
|
In summary, preview1 was changed retrospectively to require dot entries and
|
|||
|
preview2 was changed to require their absence. This rule was reverse engineered
|
|||
|
from wasmtime tests, and affirmed on two false premises:
|
|||
|
|
|||
|
* POSIX compliance requires dot entries
|
|||
|
* POSIX literally says these are optional
|
|||
|
* WASI cannot make changes because there are existing P1 programs.
|
|||
|
* Changes to Preview 1 happened before and after this topic.
|
|||
|
|
|||
|
As of June 2023, wasi-testsuite still only runs on Linux, so compliance of this
|
|||
|
rule on Windows is left to runtimes to decide to validate. The preview2 adapter
|
|||
|
uses fake cookies zero and one to refer to dot dirents, uses a real inode for
|
|||
|
the dot(".") entry and zero inode for dot-dot("..").
|
|||
|
|
|||
|
See https://github.com/WebAssembly/wasi-filesystem/issues/3
|
|||
|
See https://github.com/WebAssembly/WASI/tree/snapshot-01
|
|||
|
See https://github.com/WebAssembly/WASI/issues/9
|
|||
|
See https://github.com/WebAssembly/WASI/pull/458
|
|||
|
See https://github.com/WebAssembly/wasi-testsuite/pull/32
|
|||
|
See https://github.com/WebAssembly/wasi-libc/pull/345
|
|||
|
See https://github.com/WebAssembly/wasi-testsuite/issues/52
|
|||
|
See https://github.com/WebAssembly/WASI/pull/516
|
|||
|
See https://github.com/WebAssembly/meetings/blob/main/wasi/2023/WASI-02-09.md#should-preview1-fd_readdir-filter-out--and-
|
|||
|
See https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L1026-L1041
|
|||
|
|
|||
|
### Why are dot (".") and dot-dot ("..") entries problematic?
|
|||
|
|
|||
|
When reading a directory, dot (".") and dot-dot ("..") entries are problematic.
|
|||
|
For example, Go does not return them from `os.ReadDir`, and materializing them
|
|||
|
is complicated (at least dot-dot is).
|
|||
|
|
|||
|
A directory entry has stat information in it. The stat information includes
|
|||
|
inode which is used for comparing file equivalence. In the simple case of dot,
|
|||
|
we could materialize a special entry to expose the same info as stat on the fd
|
|||
|
would return. However, doing this and not doing dot-dot would cause confusion,
|
|||
|
and dot-dot is far more tricky. To back-fill inode information about a parent
|
|||
|
directory would be costly and subtle. For example, the pre-open (mount) of the
|
|||
|
directory may be different than its logical parent. This is easy to understand
|
|||
|
when considering the common case of mounting "/" and "/tmp" as pre-opens. To
|
|||
|
implement ".." from "/tmp" requires information from a separate pre-open, this
|
|||
|
includes state to even know the difference. There are easier edge cases as
|
|||
|
well, such as the decision to not return ".." from a root path. In any case,
|
|||
|
this should start to explain that faking entries when underlying stdlib doesn't
|
|||
|
return them is tricky and requires quite a lot of state.
|
|||
|
|
|||
|
Another issue is around the `Dirent.Off` value of a directory entry, sometimes
|
|||
|
called a "cookie" in Linux man pagers. When the host operating system or
|
|||
|
library function does not return dot entries, to support functions such as
|
|||
|
`seekdir`, you still need a value for `Dirent.Off`. Naively, you can synthesize
|
|||
|
these by choosing sequential offsets zero and one. However, POSIX strictly says
|
|||
|
offsets should be treated opaquely. The backing filesystem could use these to
|
|||
|
represent real entries. For example, a directory with one entry could use zero
|
|||
|
as the `Dirent.Off` value. If you also used zero for the "." dirent, there
|
|||
|
would be a clash. This means if you synthesize `Dirent.Off` for any entry, you
|
|||
|
need to synthesize this value for all entries. In practice, the simplest way is
|
|||
|
using an incrementing number, such as done in the WASI preview2 adapter.
|
|||
|
|
|||
|
Working around these issues causes expense to all users of wazero, so we'd
|
|||
|
then look to see if that would be justified or not. However, the most common
|
|||
|
compilers involved in end user questions, as of early 2023 are TinyGo, Rust and
|
|||
|
Zig. All of these compile code which ignores dot and dot-dot entries. In other
|
|||
|
words, faking these entries would not only cost our codebase with complexity,
|
|||
|
but it would also add unnecessary overhead as the values aren't commonly used.
|
|||
|
|
|||
|
The final reason why we might do this, is an end users or a specification
|
|||
|
requiring us to. As of early 2023, no end user has raised concern over Go and
|
|||
|
by extension wazero not returning dot and dot-dot. The snapshot-01 spec of WASI
|
|||
|
does not mention anything on this point. Also, POSIX has the following to say,
|
|||
|
which summarizes to "these are optional"
|
|||
|
|
|||
|
> The readdir() function shall not return directory entries containing empty names. If entries for dot or dot-dot exist, one entry shall be returned for dot and one entry shall be returned for dot-dot; otherwise, they shall not be returned.
|
|||
|
|
|||
|
Unfortunately, as described above, the WASI project decided in early 2023 to
|
|||
|
require dot entries in both the spec and the wasi-testsuite. For only this
|
|||
|
reason, wazero adds overhead to synthesize dot entries despite it being
|
|||
|
unnecessary for most users.
|
|||
|
|
|||
|
See https://pubs.opengroup.org/onlinepubs/9699919799/functions/readdir.html
|
|||
|
See https://github.com/golang/go/blob/go1.20/src/os/dir_unix.go#L108-L110
|
|||
|
See https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L1026-L1041
|
|||
|
|
|||
|
### Why don't we pre-populate an inode for the dot-dot ("..") entry?
|
|||
|
|
|||
|
We only populate an inode for dot (".") because wasi-testsuite requires it, and
|
|||
|
we likely already have it (because we cache it). We could attempt to populate
|
|||
|
one for dot-dot (".."), but chose not to.
|
|||
|
|
|||
|
Firstly, wasi-testsuite does not require the inode of dot-dot, possibly because
|
|||
|
the wasip2 adapter doesn't populate it (but we don't really know why).
|
|||
|
|
|||
|
The only other reason to populate it would be to avoid wasi-libc's stat fanout
|
|||
|
when it is missing. However, wasi-libc explicitly doesn't fan-out to lstat on
|
|||
|
the ".." entry on a zero ino.
|
|||
|
|
|||
|
Fetching dot-dot's inode despite the above not only doesn't help wasi-libc, but
|
|||
|
it also hurts languages that don't use it, such as Go. These languages would
|
|||
|
pay a stat syscall penalty even if they don't need the inode. In fact, Go
|
|||
|
discards both dot entries!
|
|||
|
|
|||
|
In summary, there are no significant upsides in attempting to pre-fetch
|
|||
|
dot-dot's inode, and there are downsides to doing it anyway.
|
|||
|
|
|||
|
See
|
|||
|
* https://github.com/WebAssembly/wasi-libc/blob/bd950eb128bff337153de217b11270f948d04bb4/libc-bottom-half/cloudlibc/src/libc/dirent/readdir.c#L87-L94
|
|||
|
* https://github.com/WebAssembly/wasi-testsuite/blob/main/tests/rust/src/bin/fd_readdir.rs#L108
|
|||
|
* https://github.com/bytecodealliance/preview2-prototyping/blob/e4c04bcfbd11c42c27c28984948d501a3e168121/crates/wasi-preview1-component-adapter/src/lib.rs#L1037
|
|||
|
|
|||
|
### Why don't we require inodes to be non-zero?
|
|||
|
|
|||
|
We don't require a non-zero value for `Dirent.Ino` because doing so can prevent
|
|||
|
a real one from resolving later via `Stat_t.Ino`.
|
|||
|
|
|||
|
We define `Ino` like `d_ino` in POSIX which doesn't special-case zero. It can
|
|||
|
be zero for a few reasons:
|
|||
|
|
|||
|
* The file is not a regular file or directory.
|
|||
|
* The underlying filesystem does not support inodes. e.g. embed:fs
|
|||
|
* A directory doesn't include inodes, but a later stat can. e.g. Windows
|
|||
|
* The backend is based on wasi-filesystem (a.k.a wasip2), which has
|
|||
|
`directory_entry.inode` optional, and might remove it entirely.
|
|||
|
|
|||
|
There are other downsides to returning a zero inode in widely used compilers:
|
|||
|
|
|||
|
* File equivalence utilities, like `os.SameFile` will not work.
|
|||
|
* wasi-libc's `wasip1` mode will call `lstat` and attempt to retrieve a
|
|||
|
non-zero value (unless the entry is named "..").
|
|||
|
|
|||
|
A new compiler may accidentally skip a `Dirent` with a zero `Ino` if emulating
|
|||
|
a non-POSIX function and re-using `Dirent.Ino` for `d_fileno`.
|
|||
|
|
|||
|
* Linux `getdents` doesn't define `d_fileno` must be non-zero
|
|||
|
* BSD `getdirentries` is implementation specific. For example, OpenBSD will
|
|||
|
return dirents with a zero `d_fileno`, but Darwin will skip them.
|
|||
|
|
|||
|
The above shouldn't be a problem, even in the case of BSD, because `wasip1` is
|
|||
|
defined more in terms of `getdents` than `getdirentries`. The bottom half of
|
|||
|
either should treat `wasip1` (or any similar ABI such as wasix or wasip2) as a
|
|||
|
different operating system and either use different logic that doesn't skip, or
|
|||
|
synthesize a fake non-zero `d_fileno` when `d_ino` is zero.
|
|||
|
|
|||
|
However, this has been a problem. Go's `syscall.ParseDirent` utility is shared
|
|||
|
for all `GOOS=unix`. For simplicity, this abstracts `direntIno` with data from
|
|||
|
`d_fileno` or `d_ino`, and drops if either are zero, even if `d_fileno` is the
|
|||
|
only field with zero explicitly defined. This led to a change to special case
|
|||
|
`GOOS=wasip1` as otherwise virtual files would be unconditionally skipped.
|
|||
|
|
|||
|
In practice, this problem is rather unique due to so many compilers relying on
|
|||
|
wasi-libc, which tolerates a zero inode. For example, while issues were
|
|||
|
reported about the performance regression when wasi-libc began doing a fan-out
|
|||
|
on zero `Dirent.Ino`, no issues were reported about dirents being dropped as a
|
|||
|
result.
|
|||
|
|
|||
|
In summary, rather than complicating implementation and forcing non-zero inodes
|
|||
|
for a rare case, we permit zero. We instead document this topic thoroughly, so
|
|||
|
that emerging compilers can re-use the research and reference it on conflict.
|
|||
|
We also document that `Ino` should be non-zero, so that users implementing that
|
|||
|
field will attempt to get it.
|
|||
|
|
|||
|
See
|
|||
|
* https://github.com/WebAssembly/wasi-filesystem/pull/81
|
|||
|
* https://github.com/WebAssembly/wasi-libc/blob/bd950eb128bff337153de217b11270f948d04bb4/libc-bottom-half/cloudlibc/src/libc/dirent/readdir.c#L87-L94
|
|||
|
* https://linux.die.net/man/3/getdents
|
|||
|
* https://www.unix.com/man-page/osx/2/getdirentries/
|
|||
|
* https://man.openbsd.org/OpenBSD-5.4/getdirentries.2
|
|||
|
* https://github.com/golang/go/blob/go1.20/src/syscall/dirent.go#L60-L102
|
|||
|
* https://go-review.googlesource.com/c/go/+/507915
|
|||
|
|
|||
|
## sys.Walltime and Nanotime
|
|||
|
|
|||
|
The `sys` package has two function types, `Walltime` and `Nanotime` for real
|
|||
|
and monotonic clock exports. The naming matches conventions used in Go.
|
|||
|
|
|||
|
```go
|
|||
|
func time_now() (sec int64, nsec int32, mono int64) {
|
|||
|
sec, nsec = walltime()
|
|||
|
return sec, nsec, nanotime()
|
|||
|
}
|
|||
|
```
|
|||
|
|
|||
|
Splitting functions for wall and clock time allow implementations to choose
|
|||
|
whether to implement the clock once (as in Go), or split them out.
|
|||
|
|
|||
|
Each can be configured with a `ClockResolution`, although is it usually
|
|||
|
incorrect as detailed in a sub-heading below. The only reason for exposing this
|
|||
|
is to satisfy WASI:
|
|||
|
|
|||
|
See https://github.com/WebAssembly/wasi-clocks
|
|||
|
|
|||
|
### Why default to fake time?
|
|||
|
|
|||
|
WebAssembly has an implicit design pattern of capabilities based security. By
|
|||
|
defaulting to a fake time, we reduce the chance of timing attacks, at the cost
|
|||
|
of requiring configuration to opt-into real clocks.
|
|||
|
|
|||
|
See https://gruss.cc/files/fantastictimers.pdf for an example attacks.
|
|||
|
|
|||
|
### Why does fake time increase on reading?
|
|||
|
|
|||
|
Both the fake nanotime and walltime increase by 1ms on reading. Particularly in
|
|||
|
the case of nanotime, this prevents spinning.
|
|||
|
|
|||
|
### Why not `time.Clock`?
|
|||
|
|
|||
|
wazero can't use `time.Clock` as a plugin for clock implementation as it is
|
|||
|
only substitutable with build flags (`faketime`) and conflates wall and
|
|||
|
monotonic time in the same call.
|
|||
|
|
|||
|
Go's `time.Clock` was added monotonic time after the fact. For portability with
|
|||
|
prior APIs, a decision was made to combine readings into the same API call.
|
|||
|
|
|||
|
See https://go.googlesource.com/proposal/+/master/design/12914-monotonic.md
|
|||
|
|
|||
|
WebAssembly time imports do not have the same concern. In fact even Go's
|
|||
|
imports for clocks split walltime from nanotime readings.
|
|||
|
|
|||
|
See https://github.com/golang/go/blob/go1.20/misc/wasm/wasm_exec.js#L243-L255
|
|||
|
|
|||
|
Finally, Go's clock is not an interface. WebAssembly users who want determinism
|
|||
|
or security need to be able to substitute an alternative clock implementation
|
|||
|
from the host process one.
|
|||
|
|
|||
|
### `ClockResolution`
|
|||
|
|
|||
|
A clock's resolution is hardware and OS dependent so requires a system call to retrieve an accurate value.
|
|||
|
Go does not provide a function for getting resolution, so without CGO we don't have an easy way to get an actual
|
|||
|
value. For now, we return fixed values of 1us for realtime and 1ns for monotonic, assuming that realtime clocks are
|
|||
|
often lower precision than monotonic clocks. In the future, this could be improved by having OS+arch specific assembly
|
|||
|
to make syscalls.
|
|||
|
|
|||
|
For example, Go implements time.Now for linux-amd64 with this [assembly](https://github.com/golang/go/blob/go1.20/src/runtime/time_linux_amd64.s).
|
|||
|
Because retrieving resolution is not generally called often, unlike getting time, it could be appropriate to only
|
|||
|
implement the fallback logic that does not use VDSO (executing syscalls in user mode). The syscall for clock_getres
|
|||
|
is 229 and should be usable. https://pkg.go.dev/syscall#pkg-constants.
|
|||
|
|
|||
|
If implementing similar for Windows, [mingw](https://github.com/mirror/mingw-w64/blob/6a0e9165008f731bccadfc41a59719cf7c8efc02/mingw-w64-libraries/winpthreads/src/clock.c#L77
|
|||
|
) is often a good source to find the Windows API calls that correspond
|
|||
|
to a POSIX method.
|
|||
|
|
|||
|
Writing assembly would allow making syscalls without CGO, but comes with the cost that it will require implementations
|
|||
|
across many combinations of OS and architecture.
|
|||
|
|
|||
|
## sys.Nanosleep
|
|||
|
|
|||
|
All major programming languages have a `sleep` mechanism to block for a
|
|||
|
duration. Sleep is typically implemented by a WASI `poll_oneoff` relative clock
|
|||
|
subscription.
|
|||
|
|
|||
|
For example, the below ends up calling `wasi_snapshot_preview1.poll_oneoff`:
|
|||
|
|
|||
|
```zig
|
|||
|
const std = @import("std");
|
|||
|
pub fn main() !void {
|
|||
|
std.time.sleep(std.time.ns_per_s * 5);
|
|||
|
}
|
|||
|
```
|
|||
|
|
|||
|
Besides Zig, this is also the case with TinyGo (`-target=wasi`) and Rust
|
|||
|
(`--target wasm32-wasi`).
|
|||
|
|
|||
|
We decided to expose `sys.Nanosleep` to allow overriding the implementation
|
|||
|
used in the common case, even if it isn't used by Go, because this gives an
|
|||
|
easy and efficient closure over a common program function. We also documented
|
|||
|
`sys.Nanotime` to warn users that some compilers don't optimize sleep.
|
|||
|
|
|||
|
## sys.Osyield
|
|||
|
|
|||
|
We expose `sys.Osyield`, to allow users to control the behavior of WASI's
|
|||
|
`sched_yield` without a new build of wazero. This is mainly for parity with
|
|||
|
all other related features which we allow users to implement, including
|
|||
|
`sys.Nanosleep`. Unlike others, we don't provide an out-of-box implementation
|
|||
|
primarily because it will cause performance problems when accessed.
|
|||
|
|
|||
|
For example, the below implementation uses CGO, which might result in a 1us
|
|||
|
delay per invocation depending on the platform.
|
|||
|
|
|||
|
See https://github.com/golang/go/issues/19409#issuecomment-284788196
|
|||
|
```go
|
|||
|
//go:noescape
|
|||
|
//go:linkname osyield runtime.osyield
|
|||
|
func osyield()
|
|||
|
```
|
|||
|
|
|||
|
In practice, a request to customize this is unlikely to happen until other
|
|||
|
thread based functions are implemented. That said, as of early 2023, there are
|
|||
|
a few signs of implementation interest and cross-referencing:
|
|||
|
|
|||
|
See https://github.com/WebAssembly/stack-switching/discussions/38
|
|||
|
See https://github.com/WebAssembly/wasi-threads#what-can-be-skipped
|
|||
|
See https://slinkydeveloper.com/Kubernetes-controllers-A-New-Hope/
|
|||
|
|
|||
|
## sys.Stat_t
|
|||
|
|
|||
|
We expose `stat` information as `sys.Stat_t`, like `syscall.Stat_t` except
|
|||
|
defined without build constraints. For example, you can use `sys.Stat_t` on
|
|||
|
`GOOS=windows` which doesn't define `syscall.Stat_t`.
|
|||
|
|
|||
|
The first use case of this is to return inodes from `fs.FileInfo` without
|
|||
|
relying on platform-specifics. For example, a user could return `*sys.Stat_t`
|
|||
|
from `info.Sys()` and define a non-zero inode for a virtual file, or map a
|
|||
|
real inode to a virtual one.
|
|||
|
|
|||
|
Notable choices per field are listed below, where `sys.Stat_t` is unlike
|
|||
|
`syscall.Stat_t` on `GOOS=linux`, or needs clarification. One common issue
|
|||
|
not repeated below is that numeric fields are 64-bit when at least one platform
|
|||
|
defines it that large. Also, zero values are equivalent to nil or absent.
|
|||
|
|
|||
|
* `Dev` and `Ino` (`Inode`) are both defined unsigned as they are defined
|
|||
|
opaque, and most `syscall.Stat_t` also defined them unsigned. There are
|
|||
|
separate sections in this document discussing the impact of zero in `Ino`.
|
|||
|
* `Mode` is defined as a `fs.FileMode` even though that is not defined in POSIX
|
|||
|
and will not map to all possible values. This is because the current use is
|
|||
|
WASI, which doesn't define any types or features not already supported. By
|
|||
|
using `fs.FileMode`, we can re-use routine experience in Go.
|
|||
|
* `NLink` is unsigned because it is defined that way in `syscall.Stat_t`: there
|
|||
|
can never be less than zero links to a file. We suggest defaulting to 1 in
|
|||
|
conversions when information is not knowable because at least that many links
|
|||
|
exist.
|
|||
|
* `Size` is signed because it is defined that way in `syscall.Stat_t`: while
|
|||
|
regular files and directories will always be non-negative, irregular files
|
|||
|
are possibly negative or not defined. Notably sparse files are known to
|
|||
|
return negative values.
|
|||
|
* `Atim`, `Mtim` and `Ctim` are signed because they are defined that way in
|
|||
|
`syscall.Stat_t`: Negative values are time before 1970. The resolution is
|
|||
|
nanosecond because that's the maximum resolution currently supported in Go.
|
|||
|
|
|||
|
### Why do we use `sys.EpochNanos` instead of `time.Time` or similar?
|
|||
|
|
|||
|
To simplify documentation, we defined a type alias `sys.EpochNanos` for int64.
|
|||
|
`time.Time` is a data structure, and we could have used this for
|
|||
|
`syscall.Stat_t` time values. The most important reason we do not is conversion
|
|||
|
penalty deriving time from common types.
|
|||
|
|
|||
|
The most common ABI used in `wasip2`. This, and compatible ABI such as `wasix`,
|
|||
|
encode timestamps in memory as a 64-bit number. If we used `time.Time`, we
|
|||
|
would have to convert an underlying type like `syscall.Timespec` to `time.Time`
|
|||
|
only to later have to call `.UnixNano()` to convert it back to a 64-bit number.
|
|||
|
|
|||
|
In the future, the component model module "wasi-filesystem" may represent stat
|
|||
|
timestamps with a type shared with "wasi-clocks", abstractly structured similar
|
|||
|
to `time.Time`. However, component model intentionally does not define an ABI.
|
|||
|
It is likely that the canonical ABI for timestamp will be in two parts, but it
|
|||
|
is not required for it to be intermediately represented this way. A utility
|
|||
|
like `syscall.NsecToTimespec` could split an int64 so that it could be written
|
|||
|
to memory as 96 bytes (int64, int32), without allocating a struct.
|
|||
|
|
|||
|
Finally, some may confuse epoch nanoseconds with 32-bit epoch seconds. While
|
|||
|
32-bit epoch seconds has "The year 2038" problem, epoch nanoseconds has
|
|||
|
"The Year 2262" problem, which is even less concerning for this library. If
|
|||
|
the Go programming language and wazero exist in the 2200's, we can make a major
|
|||
|
version increment to adjust the `sys.EpochNanos` approach. Meanwhile, we have
|
|||
|
faster code.
|
|||
|
|
|||
|
## poll_oneoff
|
|||
|
|
|||
|
`poll_oneoff` is a WASI API for waiting for I/O events on multiple handles.
|
|||
|
It is conceptually similar to the POSIX `poll(2)` syscall.
|
|||
|
The name is not `poll`, because it references [“the fact that this function is not efficient
|
|||
|
when used repeatedly with the same large set of handles”][poll_oneoff].
|
|||
|
|
|||
|
We chose to support this API in a handful of cases that work for regular files
|
|||
|
and standard input. We currently do not support other types of file descriptors such
|
|||
|
as socket handles.
|
|||
|
|
|||
|
### Clock Subscriptions
|
|||
|
|
|||
|
As detailed above in [sys.Nanosleep](#sysnanosleep), `poll_oneoff` handles
|
|||
|
relative clock subscriptions. In our implementation we use `sys.Nanosleep()`
|
|||
|
for this purpose in most cases, except when polling for interactive input
|
|||
|
from `os.Stdin` (see more details below).
|
|||
|
|
|||
|
### FdRead and FdWrite Subscriptions
|
|||
|
|
|||
|
When subscribing a file descriptor (except `Stdin`) for reads or writes,
|
|||
|
the implementation will generally return immediately with success, unless
|
|||
|
the file descriptor is unknown. The file descriptor is not checked further
|
|||
|
for new incoming data. Any timeout is cancelled, and the API call is able
|
|||
|
to return, unless there are subscriptions to `Stdin`: these are handled
|
|||
|
separately.
|
|||
|
|
|||
|
### FdRead and FdWrite Subscription to Stdin
|
|||
|
|
|||
|
Subscribing `Stdin` for reads (writes make no sense and cause an error),
|
|||
|
requires extra care: wazero allows to configure a custom reader for `Stdin`.
|
|||
|
|
|||
|
In general, if a custom reader is found, the behavior will be the same
|
|||
|
as for regular file descriptors: data is assumed to be present and
|
|||
|
a success is written back to the result buffer.
|
|||
|
|
|||
|
However, if the reader is detected to read from `os.Stdin`,
|
|||
|
a special code path is followed, invoking `sysfs.poll()`.
|
|||
|
|
|||
|
`sysfs.poll()` is a wrapper for `poll(2)` on POSIX systems,
|
|||
|
and it is emulated on Windows.
|
|||
|
|
|||
|
### Poll on POSIX
|
|||
|
|
|||
|
On POSIX systems, `poll(2)` allows to wait for incoming data on a file
|
|||
|
descriptor, and block until either data becomes available or the timeout
|
|||
|
expires.
|
|||
|
|
|||
|
Usage of `syfs.poll()` is currently only reserved for standard input, because
|
|||
|
|
|||
|
1. it is really only necessary to handle interactive input: otherwise,
|
|||
|
there is no way in Go to peek from Standard Input without actually
|
|||
|
reading (and thus consuming) from it;
|
|||
|
|
|||
|
2. if `Stdin` is connected to a pipe, it is ok in most cases to return
|
|||
|
with success immediately;
|
|||
|
|
|||
|
3. `syfs.poll()` is currently a blocking call, irrespective of goroutines,
|
|||
|
because the underlying syscall is; thus, it is better to limit its usage.
|
|||
|
|
|||
|
So, if the subscription is for `os.Stdin` and the handle is detected
|
|||
|
to correspond to an interactive session, then `sysfs.poll()` will be
|
|||
|
invoked with a the `Stdin` handle *and* the timeout.
|
|||
|
|
|||
|
This also means that in this specific case, the timeout is uninterruptible,
|
|||
|
unless data becomes available on `Stdin` itself.
|
|||
|
|
|||
|
### Select on Windows
|
|||
|
|
|||
|
On Windows `sysfs.poll()` cannot be delegated to a single
|
|||
|
syscall, because there is no single syscall to handle sockets,
|
|||
|
pipes and regular files.
|
|||
|
|
|||
|
Instead, we emulate its behavior for the cases that are currently
|
|||
|
of interest.
|
|||
|
|
|||
|
- For regular files, we _always_ report them as ready, as
|
|||
|
[most operating systems do anyway][async-io-windows].
|
|||
|
|
|||
|
- For pipes, we invoke [`PeekNamedPipe`][peeknamedpipe]
|
|||
|
for each file handle we detect is a pipe open for reading.
|
|||
|
We currently ignore pipes open for writing.
|
|||
|
|
|||
|
- Notably, we include also support for sockets using the [WinSock
|
|||
|
implementation of `poll`][wsapoll], but instead
|
|||
|
of relying on the timeout argument of the `WSAPoll` function,
|
|||
|
we set a 0-duration timeout so that it behaves like a peek.
|
|||
|
|
|||
|
This way, we can check for regular files all at once,
|
|||
|
at the beginning of the function, then we poll pipes and
|
|||
|
sockets periodically using a cancellable `time.Tick`,
|
|||
|
which plays nicely with the rest of the Go runtime.
|
|||
|
|
|||
|
### Impact of blocking
|
|||
|
|
|||
|
Because this is a blocking syscall, it will also block the carrier thread of
|
|||
|
the goroutine, preventing any means to support context cancellation directly.
|
|||
|
|
|||
|
There are ways to obviate this issue. We outline here one idea, that is however
|
|||
|
not currently implemented. A common approach to support context cancellation is
|
|||
|
to add a signal file descriptor to the set, e.g. the read-end of a pipe or an
|
|||
|
eventfd on Linux. When the context is canceled, we may unblock a Select call by
|
|||
|
writing to the fd, causing it to return immediately. This however requires to
|
|||
|
do a bit of housekeeping to hide the "special" FD from the end-user.
|
|||
|
|
|||
|
[poll_oneoff]: https://github.com/WebAssembly/wasi-poll#why-is-the-function-called-poll_oneoff
|
|||
|
[async-io-windows]: https://tinyclouds.org/iocp_links
|
|||
|
[peeknamedpipe]: https://learn.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-peeknamedpipe
|
|||
|
[wsapoll]: https://learn.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-wsapoll
|
|||
|
|
|||
|
## Signed encoding of integer global constant initializers
|
|||
|
|
|||
|
wazero treats integer global constant initializers signed as their interpretation is not known at declaration time. For
|
|||
|
example, there is no signed integer [value type](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#value-types%E2%91%A0).
|
|||
|
|
|||
|
To get at the problem, let's use an example.
|
|||
|
```
|
|||
|
(global (export "start_epoch") i64 (i64.const 1620216263544))
|
|||
|
```
|
|||
|
|
|||
|
In both signed and unsigned LEB128 encoding, this value is the same bit pattern. The problem is that some numbers are
|
|||
|
not. For example, 16256 is `807f` encoded as unsigned, but `80ff00` encoded as signed.
|
|||
|
|
|||
|
While the specification mentions uninterpreted integers are in abstract [unsigned values](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#integers%E2%91%A0),
|
|||
|
the binary encoding is clear that they are encoded [signed](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#integers%E2%91%A4).
|
|||
|
|
|||
|
For consistency, we go with signed encoding in the special case of global constant initializers.
|
|||
|
|
|||
|
## Implementation limitations
|
|||
|
|
|||
|
WebAssembly 1.0 (20191205) specification allows runtimes to [limit certain aspects of Wasm module or execution](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#a2-implementation-limitations).
|
|||
|
|
|||
|
wazero limitations are imposed pragmatically and described below.
|
|||
|
|
|||
|
### Number of functions in a module
|
|||
|
|
|||
|
The possible number of function instances in [a module](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#module-instances%E2%91%A0) is not specified in the WebAssembly specifications since [`funcaddr`](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#syntax-funcaddr) corresponding to a function instance in a store can be arbitrary number.
|
|||
|
wazero limits the maximum function instances to 2^27 as even that number would occupy 1GB in function pointers.
|
|||
|
|
|||
|
That is because not only we _believe_ that all use cases are fine with the limitation, but also we have no way to test wazero runtimes under these unusual circumstances.
|
|||
|
|
|||
|
### Number of function types in a store
|
|||
|
|
|||
|
There's no limitation on the number of function types in [a store](https://www.w3.org/TR/2019/REC-wasm-core-1-20191205/#store%E2%91%A0) according to the spec. In wazero implementation, we assign each function type to a unique ID, and choose to use `uint32` to represent the IDs.
|
|||
|
Therefore the maximum number of function types a store can have is limited to 2^27 as even that number would occupy 512MB just to reference the function types.
|
|||
|
|
|||
|
This is due to the same reason for the limitation on the number of functions above.
|
|||
|
|
|||
|
### Number of values on the stack in a function
|
|||
|
|
|||
|
While the the spec does not clarify a limitation of function stack values, wazero limits this to 2^27 = 134,217,728.
|
|||
|
The reason is that we internally represent all the values as 64-bit integers regardless of its types (including f32, f64), and 2^27 values means
|
|||
|
1 GiB = (2^30). 1 GiB is the reasonable for most applications [as we see a Goroutine has 250 MB as a limit on the stack for 32-bit arch](https://github.com/golang/go/blob/go1.20/src/runtime/proc.go#L152-L159), considering that WebAssembly is (currently) 32-bit environment.
|
|||
|
|
|||
|
All the functions are statically analyzed at module instantiation phase, and if a function can potentially reach this limit, an error is returned.
|
|||
|
|
|||
|
### Number of globals in a module
|
|||
|
|
|||
|
Theoretically, a module can declare globals (including imports) up to 2^32 times. However, wazero limits this to 2^27(134,217,728) per module.
|
|||
|
That is because internally we store globals in a slice with pointer types (meaning 8 bytes on 64-bit platforms), and therefore 2^27 globals
|
|||
|
means that we have 1 GiB size of slice which seems large enough for most applications.
|
|||
|
|
|||
|
### Number of tables in a module
|
|||
|
|
|||
|
While the the spec says that a module can have up to 2^32 tables, wazero limits this to 2^27 = 134,217,728.
|
|||
|
One of the reasons is even that number would occupy 1GB in the pointers tables alone. Not only that, we access tables slice by
|
|||
|
table index by using 32-bit signed offset in the compiler implementation, which means that the table index of 2^27 can reach 2^27 * 8 (pointer size on 64-bit machines) = 2^30 offsets in bytes.
|
|||
|
|
|||
|
We _believe_ that all use cases are fine with the limitation, but also note that we have no way to test wazero runtimes under these unusual circumstances.
|
|||
|
|
|||
|
If a module reaches this limit, an error is returned at the compilation phase.
|
|||
|
|
|||
|
## Compiler engine implementation
|
|||
|
|
|||
|
### Why it's safe to execute runtime-generated machine codes against async Goroutine preemption
|
|||
|
|
|||
|
Goroutine preemption is the mechanism of the Go runtime to switch goroutines contexts on an OS thread.
|
|||
|
There are two types of preemption: cooperative preemption and async preemption. The former happens, for example,
|
|||
|
when making a function call, and it is not an issue for our runtime-generated functions as they do not make
|
|||
|
direct function calls to Go-implemented functions. On the other hand, the latter, async preemption, can be problematic
|
|||
|
since it tries to interrupt the execution of Goroutine at any point of function, and manipulates CPU register states.
|
|||
|
|
|||
|
Fortunately, our runtime-generated machine codes do not need to take the async preemption into account.
|
|||
|
All the assembly codes are entered via the trampoline implemented as Go Assembler Function (e.g. [arch_amd64.s](./arch_amd64.s)),
|
|||
|
and as of Go 1.20, these assembler functions are considered as _unsafe_ for async preemption:
|
|||
|
- https://github.com/golang/go/blob/go1.20rc1/src/runtime/preempt.go#L406-L407
|
|||
|
- https://github.com/golang/go/blob/9f0234214473dfb785a5ad84a8fc62a6a395cbc3/src/runtime/traceback.go#L227
|
|||
|
|
|||
|
From the Go runtime point of view, the execution of runtime-generated machine codes is considered as a part of
|
|||
|
that trampoline function. Therefore, runtime-generated machine code is also correctly considered unsafe for async preemption.
|
|||
|
|
|||
|
## Why context cancellation is handled in Go code rather than native code
|
|||
|
|
|||
|
Since [wazero v1.0.0-pre.9](https://github.com/tetratelabs/wazero/releases/tag/v1.0.0-pre.9), the runtime
|
|||
|
supports integration with Go contexts to interrupt execution after a timeout, or in response to explicit cancellation.
|
|||
|
This support is internally implemented as a special opcode `builtinFunctionCheckExitCode` that triggers the execution of
|
|||
|
a Go function (`ModuleInstance.FailIfClosed`) that atomically checks a sentinel value at strategic points in the code.
|
|||
|
|
|||
|
[It _is indeed_ possible to check the sentinel value directly, without leaving the native world][native_check], thus sparing some cycles;
|
|||
|
however, because native code never preempts (see section above), this may lead to a state where the other goroutines
|
|||
|
never get the chance to run, and thus never get the chance to set the sentinel value; effectively preventing
|
|||
|
cancellation from taking place.
|
|||
|
|
|||
|
[native_check]: https://github.com/tetratelabs/wazero/issues/1409
|
|||
|
|
|||
|
## Golang patterns
|
|||
|
|
|||
|
### Hammer tests
|
|||
|
Code that uses concurrency primitives, such as locks or atomics, should include "hammer tests", which run large loops
|
|||
|
inside a bounded amount of goroutines, run by half that many `GOMAXPROCS`. These are named consistently "hammer", so
|
|||
|
they are easy to find. The name inherits from some existing tests in [golang/go](https://github.com/golang/go/search?q=hammer&type=code).
|
|||
|
|
|||
|
Here is an annotated description of the key pieces of a hammer test:
|
|||
|
1. `P` declares the count of goroutines to use, defaulting to 8 or 4 if `testing.Short`.
|
|||
|
* Half this amount are the cores used, and 4 is less than a modern laptop's CPU. This allows multiple "hammer" tests to run in parallel.
|
|||
|
2. `N` declares the scale of work (loop) per goroutine, defaulting to value that finishes in ~0.1s on a modern laptop.
|
|||
|
* When in doubt, try 1000 or 100 if `testing.Short`
|
|||
|
* Remember, there are multiple hammer tests and CI nodes are slow. Slower tests hurt feedback loops.
|
|||
|
3. `defer runtime.GOMAXPROCS(runtime.GOMAXPROCS(P/2))` makes goroutines switch cores, testing visibility of shared data.
|
|||
|
4. To ensure goroutines execute at the same time, block them with `sync.WaitGroup`, initialized to `Add(P)`.
|
|||
|
* `sync.WaitGroup` internally uses `runtime_Semacquire` not available in any other library.
|
|||
|
* `sync.WaitGroup.Add` with a negative value can unblock many goroutines at the same time, e.g. without a for loop.
|
|||
|
5. Track goroutines progress via `finished := make(chan int)` where each goroutine in `P` defers `finished <- 1`.
|
|||
|
1. Tests use `require.XXX`, so `recover()` into `t.Fail` in a `defer` function before `finished <- 1`.
|
|||
|
* This makes it easier to spot larger concurrency problems as you see each failure, not just the first.
|
|||
|
2. After the `defer` function, await unblocked, then run the stateful function `N` times in a normal loop.
|
|||
|
* This loop should trigger shared state problems as locks or atomics are contended by `P` goroutines.
|
|||
|
6. After all `P` goroutines launch, atomically release all of them with `WaitGroup.Add(-P)`.
|
|||
|
7. Block the runner on goroutine completion, by (`<-finished`) for each `P`.
|
|||
|
8. When all goroutines complete, `return` if `t.Failed()`, otherwise perform follow-up state checks.
|
|||
|
|
|||
|
This is implemented in wazero in [hammer.go](internal/testing/hammer/hammer.go)
|
|||
|
|
|||
|
### Lock-free, cross-goroutine observations of updates
|
|||
|
|
|||
|
How to achieve cross-goroutine reads of a variable are not explicitly defined in https://go.dev/ref/mem. wazero uses
|
|||
|
atomics to implement this following unofficial practice. For example, a `Close` operation can be guarded to happen only
|
|||
|
once via compare-and-swap (CAS) against a zero value. When we use this pattern, we consistently use atomics to both
|
|||
|
read and update the same numeric field.
|
|||
|
|
|||
|
In lieu of formal documentation, we infer this pattern works from other sources (besides tests):
|
|||
|
* `sync.WaitGroup` by definition must support calling `Add` from other goroutines. Internally, it uses atomics.
|
|||
|
* rsc in golang/go#5045 writes "atomics guarantee sequential consistency among the atomic variables".
|
|||
|
|
|||
|
See https://github.com/golang/go/blob/go1.20/src/sync/waitgroup.go#L64
|
|||
|
See https://github.com/golang/go/issues/5045#issuecomment-252730563
|
|||
|
See https://www.youtube.com/watch?v=VmrEG-3bWyM
|