Skip to content

perf: ZTS: move AG and SCNG into native __thread storage, replay of #22231 #22595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
henderkes wants to merge 1 commit into
base: master
Choose a base branch
from
Open

perf: ZTS: move AG and SCNG into native __thread storage, replay of #22231 #22595

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 57 additions & 5 deletions TSRM/TSRM.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ typedef struct {
ts_allocate_ctor ctor;
ts_allocate_dtor dtor;
ptrdiff_t fast_offset;
/* When set, storage comes from __thread memory instead of being allocated by TSRM. */
void *(*tls_addr)(void);
int done;
} tsrm_resource_type;

Expand Down Expand Up @@ -164,14 +166,20 @@ TSRM_API bool tsrm_startup(int expected_threads, int expected_resources, int deb

static void ts_free_resources(tsrm_tls_entry *thread_resources)
{
bool own_thread = thread_resources->thread_id == tsrm_thread_id();

/* Need to destroy in reverse order to respect dependencies. */
for (int i = thread_resources->count - 1; i >= 0; i--) {
if (!resource_types_table[i].done) {
/* A __thread block of a foreign thread is inaccessible. */
if (resource_types_table[i].tls_addr && !own_thread) {
continue;
}

@arnaud-lb arnaud-lb Jul 6, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/* A __thread block of a foreign thread is inaccessible. */

Is that because the thread may not exist anymore, or is there another reason?

We need to ensure that shutdown_memory_manager(CG(unclean_shutdown), 1); is called in php_module_shutdown. Currently we avoid calling it in ZTS builds because we rely on the TSRM dtor, I believe.

I would go as far as removing the dtor argument from ts_allocate_tls_id since it's not going to be called for most threads during tsrm_shutdown().

@henderkes henderkes Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that because the thread may not exist anymore, or is there another reason?

It's pointing to __thread memory of another thread, but the only way it got here is if a foreign thread already died, so we'd be accessing dangling memory. It's technically accessible but there's no telling what running the dtor would lead to.

We need to ensure that shutdown_memory_manager(CG(unclean_shutdown), 1); is called in php_module_shutdown. Currently we avoid calling it in ZTS builds because we rely on the TSRM dtor, I believe.

Is this a general request? I don't change the main thread shutdown path here (it gets dtor defined and is guaranteed to run its own thread). If it's just a general request, I would postpone that to a follow-up PR.

I would go as far as removing the dtor argument from ts_allocate_tls_id since it's not going to be called for most threads during tsrm_shutdown().

Can't currently do that for AG because it does pass and run the dtor.

@arnaud-lb arnaud-lb Jul 8, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dtor does run but only for the thread that calls tsrm_shutdown() and not for other threads. Before your changes, the dtor was called for all threads during tsrm_shutdown().

@henderkes henderkes Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dtor does run but only for the thread that calls tsrm_shutdown() and not for other threads. Before your changes, the dtor was called for all threads during tsrm_shutdown().

The dtor does run for all threads that exit via ts_free_thread() -> ts_free_resources(). The clean shutdown logic is that no live threads will be registered here anymore except for the main thread during tsrm_shutdown, only failed/crashed threads (for which we can no-longer safely run the dtor for).

@arnaud-lb arnaud-lb Jul 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now see that FrankenPHP calls ts_free_thread(), but other SAPIs don't: they rely on the current behavior of tsrm_shutdown() which is to cleanup all threads at once.

Let's keep the dtor argument since it's useful for SAPIs that call ts_free_thread().

Could you add a comment on tsrm_shutdown() and a note in UPGRADING.INTERNALS?

@henderkes henderkes Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait... what multi threaded sapi's don't call ts_free_thread()? Cli/fpm are single threaded and other webservers (pasir) or ext-parallel do call it.

Is this the apache handler again?

@henderkes henderkes Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is, of course it is. I don't have a good solution for it, because it doesn't seem like Apache lets you install a thread shutdown hook that we could let ts_free_thread run on. We also can't safely reach into the threads memory without risking a segfault, so the memory just has to leak.

if (resource_types_table[i].dtor) {
resource_types_table[i].dtor(thread_resources->storage[i]);
}

if (!resource_types_table[i].fast_offset) {
if (!resource_types_table[i].fast_offset && !resource_types_table[i].tls_addr) {
free(thread_resources->storage[i]);
}
}
Expand Down Expand Up @@ -258,7 +266,10 @@ static void tsrm_update_active_threads(void)

p->storage = (void *) realloc(p->storage, sizeof(void *)*id_count);
for (j=p->count; j<id_count; j++) {
if (resource_types_table[j].fast_offset) {
if (resource_types_table[j].tls_addr) {
TSRM_ASSERT(p->thread_id == tsrm_thread_id());
p->storage[j] = resource_types_table[j].tls_addr();
} else if (resource_types_table[j].fast_offset) {
p->storage[j] = (void *) (((char*)p) + resource_types_table[j].fast_offset);
} else {
p->storage[j] = (void *) malloc(resource_types_table[j].size);
Expand Down Expand Up @@ -303,6 +314,7 @@ TSRM_API ts_rsrc_id ts_allocate_id(ts_rsrc_id *rsrc_id, size_t size, ts_allocate
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].ctor = ctor;
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].dtor = dtor;
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].fast_offset = 0;
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].tls_addr = NULL;
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].done = 0;

tsrm_update_active_threads();
Expand Down Expand Up @@ -381,6 +393,7 @@ TSRM_API ts_rsrc_id ts_allocate_fast_id_at(ts_rsrc_id *rsrc_id, size_t *offset,
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].ctor = ctor;
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].dtor = dtor;
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].fast_offset = *offset;
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].tls_addr = NULL;
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].done = 0;

tsrm_update_active_threads();
Expand All @@ -390,6 +403,41 @@ TSRM_API ts_rsrc_id ts_allocate_fast_id_at(ts_rsrc_id *rsrc_id, size_t *offset,
return *rsrc_id;
}

/* allocates a resource id whose per-thread storage is a native __thread block */
TSRM_API ts_rsrc_id ts_allocate_tls_id(ts_rsrc_id *rsrc_id, void *(*tls_addr)(void), size_t size, ts_allocate_ctor ctor, ts_allocate_dtor dtor)
{
TSRM_ERROR((TSRM_ERROR_LEVEL_CORE, "Obtaining a new TLS resource id, %d bytes", size));

tsrm_mutex_lock(tsmm_mutex);

*rsrc_id = TSRM_SHUFFLE_RSRC_ID(id_count++);

if (resource_types_table_size < id_count) {
tsrm_resource_type *_tmp;
_tmp = (tsrm_resource_type *) realloc(resource_types_table, sizeof(tsrm_resource_type)*id_count);
if (!_tmp) {
TSRM_ERROR((TSRM_ERROR_LEVEL_ERROR, "Unable to allocate storage for resource"));
*rsrc_id = 0;
tsrm_mutex_unlock(tsmm_mutex);
return 0;
}
resource_types_table = _tmp;
resource_types_table_size = id_count;
}
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].size = size;
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].ctor = ctor;
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].dtor = dtor;
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].fast_offset = 0;
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].tls_addr = tls_addr;
resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].done = 0;

tsrm_update_active_threads();
tsrm_mutex_unlock(tsmm_mutex);

TSRM_ERROR((TSRM_ERROR_LEVEL_CORE, "Successfully allocated new TLS resource id %d", *rsrc_id));
return *rsrc_id;
}

static void set_thread_local_storage_resource_to(tsrm_tls_entry *thread_resource)
{
tsrm_tls_set(thread_resource);
Expand Down Expand Up @@ -422,7 +470,9 @@ static void allocate_new_resource(tsrm_tls_entry **thread_resources_ptr, THREAD_
if (resource_types_table[i].done) {
(*thread_resources_ptr)->storage[i] = NULL;
} else {
if (resource_types_table[i].fast_offset) {
if (resource_types_table[i].tls_addr) {
(*thread_resources_ptr)->storage[i] = resource_types_table[i].tls_addr();
} else if (resource_types_table[i].fast_offset) {
(*thread_resources_ptr)->storage[i] = (void *) (((char*)(*thread_resources_ptr)) + resource_types_table[i].fast_offset);
} else {
(*thread_resources_ptr)->storage[i] = (void *) malloc(resource_types_table[i].size);
Expand Down Expand Up @@ -510,7 +560,9 @@ TSRM_API void *ts_resource_ex(ts_rsrc_id id, THREAD_T *th_id)
/* In case that extensions don't use the pointer passed from the dtor, but incorrectly
* use the global pointer, we need to setup the global pointer temporarily here. */
set_thread_local_storage_resource_to(thread_resources);
/* Free up the old resource from the old thread instance */
/* Dead thread with a recycled id: its __thread blocks are gone, and this
* thread's blocks were never constructed, so keep tls dtors from running. */
thread_resources->thread_id = 0;
ts_free_resources(thread_resources);
free((char *) thread_resources - tsrm_reserved_front);
/* Allocate a new resource at the same point in the linked list, and relink the next pointer */
Expand Down Expand Up @@ -584,7 +636,7 @@ void ts_free_id(ts_rsrc_id id)
if (resource_types_table[rsrc_id].dtor) {
resource_types_table[rsrc_id].dtor(p->storage[rsrc_id]);
}
if (!resource_types_table[rsrc_id].fast_offset) {
if (!resource_types_table[rsrc_id].fast_offset && !resource_types_table[rsrc_id].tls_addr) {
free(p->storage[rsrc_id]);
}
}
Expand Down
4 changes: 4 additions & 0 deletions TSRM/TSRM.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,10 @@ TSRM_API ts_rsrc_id ts_allocate_fast_id(ts_rsrc_id *rsrc_id, size_t *offset, siz
TSRM_API void tsrm_reserve_fast_front(size_t size);
TSRM_API ts_rsrc_id ts_allocate_fast_id_at(ts_rsrc_id *rsrc_id, size_t *offset, ptrdiff_t fixed_offset, size_t size, ts_allocate_ctor ctor, ts_allocate_dtor dtor);

/* Resource whose per-thread storage is a native __thread block.
* Must be called at startup before any other thread exists. */
TSRM_API ts_rsrc_id ts_allocate_tls_id(ts_rsrc_id *rsrc_id, void *(*tls_addr)(void), size_t size, ts_allocate_ctor ctor, ts_allocate_dtor dtor);

/* fetches the requested resource for the current thread */
TSRM_API void *ts_resource_ex(ts_rsrc_id id, THREAD_T *th_id);
#define ts_resource(id) ts_resource_ex(id, NULL)
Expand Down
6 changes: 4 additions & 2 deletions Zend/zend.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ ZEND_API int compiler_globals_id;
ZEND_API int executor_globals_id;
ZEND_API size_t compiler_globals_offset;
ZEND_API size_t executor_globals_offset;
/* ts_allocate_tls_id takes a callback so each thread resolves its own block.
* A plain &language_scanner_globals would capture only the registering thread's address. */
static void *language_scanner_globals_tls_addr(void) { return &language_scanner_globals; }
static HashTable *global_function_table = NULL;
static HashTable *global_class_table = NULL;
static HashTable *global_constants_table = NULL;
Expand Down Expand Up @@ -1021,10 +1024,9 @@ void zend_startup(zend_utility_functions *utility_functions) /* {{{ */
#ifdef ZTS
ts_allocate_fast_id_at(&compiler_globals_id, &compiler_globals_offset, ZEND_CG_OFFSET, sizeof(zend_compiler_globals), (ts_allocate_ctor) compiler_globals_ctor, (ts_allocate_dtor) compiler_globals_dtor);
ts_allocate_fast_id_at(&executor_globals_id, &executor_globals_offset, ZEND_EG_OFFSET, sizeof(zend_executor_globals), (ts_allocate_ctor) executor_globals_ctor, (ts_allocate_dtor) executor_globals_dtor);
ts_allocate_fast_id_at(&language_scanner_globals_id, &language_scanner_globals_offset, ZEND_SCNG_OFFSET, sizeof(zend_php_scanner_globals), (ts_allocate_ctor) php_scanner_globals_ctor, NULL);
ts_allocate_tls_id(&language_scanner_globals_id, language_scanner_globals_tls_addr, sizeof(zend_php_scanner_globals), (ts_allocate_ctor) php_scanner_globals_ctor, NULL);
ZEND_ASSERT(compiler_globals_offset == ZEND_CG_OFFSET);
ZEND_ASSERT(executor_globals_offset == ZEND_EG_OFFSET);
ZEND_ASSERT(language_scanner_globals_offset == ZEND_SCNG_OFFSET);
ts_allocate_fast_id(&ini_scanner_globals_id, &ini_scanner_globals_offset, sizeof(zend_ini_scanner_globals), (ts_allocate_ctor) ini_scanner_globals_ctor, NULL);
compiler_globals = ts_resource(compiler_globals_id);
executor_globals = ts_resource(executor_globals_id);
Expand Down
16 changes: 4 additions & 12 deletions Zend/zend_alloc.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2613,13 +2613,12 @@ typedef struct _zend_alloc_globals {

#ifdef ZTS
static int alloc_globals_id;
static size_t alloc_globals_offset;
# define ZEND_AG_OFFSET (ZEND_SCNG_OFFSET - (ptrdiff_t) TSRM_ALIGNED_SIZE(sizeof(zend_alloc_globals)))
# define AG(v) ZEND_TSRMG_FAST(ZEND_AG_OFFSET, zend_alloc_globals *, v)
static TSRM_TLS TSRM_TLS_MODEL_ATTR zend_alloc_globals alloc_globals;
static void *alloc_globals_tls_addr(void) { return &alloc_globals; }
#else
# define AG(v) (alloc_globals.v)
static zend_alloc_globals alloc_globals;
#endif
#define AG(v) (alloc_globals.v)

ZEND_API bool is_zend_mm(void)
{
Expand Down Expand Up @@ -3336,8 +3335,7 @@ ZEND_API void start_memory_manager(void)
# endif
#endif
#ifdef ZTS
ts_allocate_fast_id_at(&alloc_globals_id, &alloc_globals_offset, ZEND_AG_OFFSET, sizeof(zend_alloc_globals), (ts_allocate_ctor) alloc_globals_ctor, (ts_allocate_dtor) alloc_globals_dtor);
ZEND_ASSERT(alloc_globals_offset == ZEND_AG_OFFSET);
ts_allocate_tls_id(&alloc_globals_id, alloc_globals_tls_addr, sizeof(zend_alloc_globals), (ts_allocate_ctor) alloc_globals_ctor, (ts_allocate_dtor) alloc_globals_dtor);
#else
alloc_globals_ctor(&alloc_globals);
#endif
Expand Down Expand Up @@ -3583,9 +3581,3 @@ ZEND_API char * __zend_strdup(const char *s)
zend_out_of_memory();
}

#ifdef ZTS
size_t zend_mm_globals_size(void)
{
return sizeof(zend_alloc_globals);
}
#endif
4 changes: 0 additions & 4 deletions Zend/zend_alloc.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -403,10 +403,6 @@ static void apc_init_heap(void)

*/

#ifdef ZTS
size_t zend_mm_globals_size(void);
#endif

END_EXTERN_C()

#endif
4 changes: 1 addition & 3 deletions Zend/zend_globals.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -330,11 +330,9 @@ struct _zend_executor_globals {
};

#ifdef ZTS
/* Compile-time offsets of the hot globals, in a reserved region just before the
* cache pointer. ZEND_AG_OFFSET is furthest, in zend_alloc.c. */
/* Compile-time offsets of the hot globals, in a reserved region just before *_tsrm_ls_cache. */
# define ZEND_CG_OFFSET (-(ptrdiff_t) TSRM_ALIGNED_SIZE(sizeof(zend_compiler_globals)))
# define ZEND_EG_OFFSET (ZEND_CG_OFFSET - (ptrdiff_t) TSRM_ALIGNED_SIZE(sizeof(zend_executor_globals)))
# define ZEND_SCNG_OFFSET (ZEND_EG_OFFSET - (ptrdiff_t) TSRM_ALIGNED_SIZE(sizeof(zend_php_scanner_globals)))
#endif

#define EG_FLAGS_INITIAL (0)
Expand Down
14 changes: 11 additions & 3 deletions Zend/zend_globals_macros.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,12 +48,20 @@ extern ZEND_API zend_executor_globals executor_globals;

/* Language Scanner */
#ifdef ZTS
# define LANG_SCNG(v) ZEND_TSRMG_FAST(ZEND_SCNG_OFFSET, zend_php_scanner_globals *, v)
extern ZEND_API ts_rsrc_id language_scanner_globals_id;
extern ZEND_API size_t language_scanner_globals_offset;
# if defined(ZEND_WIN32) && !defined(LIBZEND_EXPORTS)
# define LANG_SCNG(v) TSRMG(language_scanner_globals_id, zend_php_scanner_globals *, v)
# else
# ifdef ZEND_WIN32
extern TSRM_TLS zend_php_scanner_globals language_scanner_globals;
# else
extern ZEND_API TSRM_TLS TSRM_TLS_MODEL_ATTR zend_php_scanner_globals language_scanner_globals;
# endif
# define LANG_SCNG(v) (language_scanner_globals.v)
# endif
#else
# define LANG_SCNG(v) (language_scanner_globals.v)
extern ZEND_API zend_php_scanner_globals language_scanner_globals;
# define LANG_SCNG(v) (language_scanner_globals.v)
#endif


Expand Down
6 changes: 5 additions & 1 deletion Zend/zend_language_scanner.l
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,11 @@
#define SCNG LANG_SCNG
#ifdef ZTS
ZEND_API ts_rsrc_id language_scanner_globals_id;
ZEND_API size_t language_scanner_globals_offset;
# ifdef ZEND_WIN32
TSRM_TLS zend_php_scanner_globals language_scanner_globals;
# else
ZEND_API TSRM_TLS TSRM_TLS_MODEL_ATTR zend_php_scanner_globals language_scanner_globals;
# endif
#else
ZEND_API zend_php_scanner_globals language_scanner_globals;
#endif
Expand Down
6 changes: 1 addition & 5 deletions main/main.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2819,13 +2819,11 @@ PHPAPI void php_reserve_tsrm_memory(void)
tsrm_reserve(
TSRM_ALIGNED_SIZE(sizeof(zend_compiler_globals)) +
TSRM_ALIGNED_SIZE(sizeof(zend_executor_globals)) +
TSRM_ALIGNED_SIZE(sizeof(zend_php_scanner_globals)) +
TSRM_ALIGNED_SIZE(sizeof(zend_ini_scanner_globals)) +
TSRM_ALIGNED_SIZE(sizeof(virtual_cwd_globals)) +
#ifdef ZEND_SIGNALS
TSRM_ALIGNED_SIZE(sizeof(zend_signal_globals_t)) +
#endif
TSRM_ALIGNED_SIZE(zend_mm_globals_size()) +
TSRM_ALIGNED_SIZE(zend_gc_globals_size()) +
TSRM_ALIGNED_SIZE(sizeof(php_core_globals)) +
TSRM_ALIGNED_SIZE(sizeof(sapi_globals_struct)) +
Expand All @@ -2845,9 +2843,7 @@ PHPAPI bool php_tsrm_startup_ex(int expected_threads)
/* Must cover the total size of every ZEND_*_OFFSET global, or the furthest underflows the block. */
tsrm_reserve_fast_front(
TSRM_ALIGNED_SIZE(sizeof(zend_compiler_globals)) +
TSRM_ALIGNED_SIZE(sizeof(zend_executor_globals)) +
TSRM_ALIGNED_SIZE(sizeof(zend_php_scanner_globals)) +
TSRM_ALIGNED_SIZE(zend_mm_globals_size())); // AG size, exposed through function call
TSRM_ALIGNED_SIZE(sizeof(zend_executor_globals)));
(void)ts_resource(0);
return ret;
}
Expand Down
Loading