jemalloc wouldn't help in the naive Python interpreter. pymalloc directly uses OS virtual memory allocation routines for its object heap, which is mmap on Linux.
I think given the nature of pymalloc (arenas and pools), it would be beneficial for CPython to provide visualization into fragmentation to understand if things like malloc_trim are actually useful (no doubt that for your workload you must have measured and seen improvements).
A moving GC is actually probably the ideal solution for Python but would be incompatible with CPython semantics (specifically that `id()` is often the pointer address in memory.)
Have you measured any improvement by doing this?
Yes, we've reduced OOMs significantly, from happening once an hour, to once every 4 - 5 hours.
jemalloc wouldn't help in the naive Python interpreter. pymalloc directly uses OS virtual memory allocation routines for its object heap, which is mmap on Linux.
I think given the nature of pymalloc (arenas and pools), it would be beneficial for CPython to provide visualization into fragmentation to understand if things like malloc_trim are actually useful (no doubt that for your workload you must have measured and seen improvements).
A moving GC is actually probably the ideal solution for Python but would be incompatible with CPython semantics (specifically that `id()` is often the pointer address in memory.)