Recursive resolving of type aliases

[mauvilsa]

typeshed_client seems the right fit for a project I am working on, though I am still trying to figure out how it is supposed to be used, and how to make it work for my case. I want to get the types of the parameters of some function. These types can have aliases in them, but I want to resolve them to basic python types or types defined in the respective python modules.

Say for example that I want the types for requests.get. First create a virtual environment:

python3.10 -m venv venv
. venv/bin/activate
pip install requests types-requests typeshed-client

Then in python:

>>> import ast
>>> import sys
>>> import typeshed_client
>>> from pathlib import Path
>>> 
>>> search_path = [Path(p) for p in sys.path]
>>> search_context = typeshed_client.get_search_context(search_path=search_path)
>>> resolver = typeshed_client.Resolver(search_context=search_context)
>>> 
>>> requests_api_get = resolver.get_fully_qualified_name('requests.api.get')
>>> print(ast.unparse(requests_api_get.ast))
def get(url: str | bytes, params: _Params | None=..., *, data: _Data | None=..., headers: _HeadersMapping | None=..., cookies: RequestsCookieJar | _TextMapping | None=..., files: _Files | None=..., auth: _Auth | None=..., timeout: _Timeout | None=..., allow_redirects: bool=..., proxies: _TextMapping | None=..., hooks: _HooksInput | None=..., stream: bool | None=..., verify: _Verify | None=..., cert: _Cert | None=..., json: Any | None=...) -> Response:
    ...

Note the type aliases _Params, _Data, _HeadersMapping, _TextMapping, _HooksInput, _Verify. I can look at the AST to figure out undefined names and then try to get them, but this seems to be something that should be part of a library:

>>> requests_api__Params = resolver.get_fully_qualified_name('requests.api._Params')
>>> print(requests_api__Params)
ImportedInfo(source_module=('requests', 'sessions'), info=NameInfo(name='_Params', is_exported=False, ast=<ast.AnnAssign object at 0x7f5112cb4f40>, child_nodes=None))
>>> print(ast.unparse(requests_api__Params.info.ast))
_Params: TypeAlias = Union[SupportsItems[_ParamsMappingKeyType, _ParamsMappingValueType], tuple[_ParamsMappingKeyType, _ParamsMappingValueType], Iterable[tuple[_ParamsMappingKeyType, _ParamsMappingValueType]], str | bytes]

Now new aliases _ParamsMappingKeyType, _ParamsMappingValueType.

>>> requests_sessions__ParamsMappingKeyType = resolver.get_fully_qualified_name('requests.sessions._ParamsMappingKeyType')
>>> print(ast.unparse(requests_sessions__ParamsMappingKeyType.ast))
_ParamsMappingKeyType: TypeAlias = str | bytes | int | float

Is there already a simpler way to achieve this? If not, would it make sense to extend typeshed_client to make this easy? One idea could be to have an option such that requests_api_get.ast already has all the type aliases are replaced with the respective ASTs. Alternatively (though less convenient) have the alias ASTs in child_nodes or in a separate element of the named tuple.

[JelleZijlstra]

What you are suggesting is indeed the current way to do it. I see how what you're asking for could be useful, but there's a couple of complexities:

• Some users may not want the type aliases replaced, because when showing documentation it may be friendlier to show the aliases instead of the expanded types.
• Sometimes we cannot expand the alias, because type aliases can be recursive.
• Blindly substituting in the AST can be unsafe because it may refer to names defined in a different module. For example, Union could be typing.Union in the source module of the alias, but ctypes.Union in the current module.

A solution could be to add a new higher-level representation of types to typeshed-client, a set of classes like the ones in https://github.com/quora/pyanalyze/blob/master/pyanalyze/value.py or https://github.com/python/mypy/blob/master/mypy/types.py.

[mauvilsa]

Thank you for the fast response. I mostly wanted to know if I was in the right track, so this is already great information. I completely agree with your points, which completely rules out replacing ASTs. But how about an additional element in the tuple (only available if requested), say called type_aliases and be a dict of ImportedInfo objects. This would already be very useful. No need to add type classes for this.

Anyway, since currently typeshed_client does not have a way to do this easier, I will implement things for what I need. And maybe mention here what I did to see if it makes sense to contribute it.

[mauvilsa]

@JelleZijlstra I implemented what I needed, see jsonargparse/_stubs_resolver.py. I am not sure if there is something that would be worth contributing back here. If you want an explanation of what the code does, let me know. If not, we could close this issue.

A couple additional comments.

• I implemented a method get_imported_info that always returns ImportedInfo or None. It was quite annoying to use get_fully_qualified_name because I had to handle possible different reponse types in every place I used it.
• I ended up doing a hack to know if a stub came from an __init__.pyi or some other pyi file. This is important to resolve imports. For example, an import from .module1 import name1 means something different if it is in parent/child.pyi (parent.child.module1?) or in parent/__init__.pyi (parent.module1). I am not sure if there was a way to figure this out without the hack that I did. Is there?