Commit 59701670 authored by Zeyi (Rice) Fan's avatar Zeyi (Rice) Fan Committed by Facebook GitHub Bot

make fb_py_win_main to dynamically find Python3.dll

Summary:
In EdenFS's latest Windows package. We are seeing DLL import errors coming from `asyncio` as it requires a system native module `_overlapped.pyd`.

The underlying cause is because when we build EdenFS CLI on Sandcastle, we are linking with Python 3.6.2. The Python36.dll shipped with the EdenFS package is also coming from that version.

However, on Windows laptop. We have Python 3.6.3. Since we are not shipping the Python system libraries with us. It uses the libraries installed in the system, and it attempts to import the `_overlapped.pyd` located at `C:\Pythone36\DLLs\`. This version is compiled against Python 3.6.3, which is incompatible with the Python36.dll we are using.

----

To resolve this, we need either ship an embedded copy of Python along with EdenFS, or teach EdenFS to use the Python distribution installed in the system. This commit tweaks the executable we prepend to the archive created with zipapp to locate `Python3.dll` dynamically. This allows us to use the Python installed in the system so we can avoid the version mismatch issue.

With this setup, we can also be shipping an embedded Python version along with EdenFS, and the Python loader can look for that path. This is demonstrated with the relative DLL loading `..\python`.

In theory, we can have a package structure like this:

```
.
├── python
│   ├── ....
│   └── python3.dll
└── bin
    ├── ...
    ├── edenfsctl.exe
    └── edenfs.exe
```

Reviewed By: xavierd

Differential Revision: D22325210

fbshipit-source-id: 96a3f9503e7865a5f9d95710ff13f019afcf04f1
parent 0c4dd654
// Copyright (c) Facebook, Inc. and its affiliates. // Copyright (c) Facebook, Inc. and its affiliates.
#define Py_LIMITED_API 1
#define WIN32_LEAN_AND_MEAN #define WIN32_LEAN_AND_MEAN
#include <Windows.h> #include <Windows.h>
#include <Python.h> #include <stdlib.h>
#include <stdio.h> #include <stdio.h>
#define PATH_SIZE 1024
typedef int (*Py_Main)(int, wchar_t**);
// Add the given path to Windows's DLL search path.
// For Windows DLL search path resolution, see:
// https://docs.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order
void add_search_path(const wchar_t* path) {
wchar_t buffer[PATH_SIZE];
wchar_t** lppPart = NULL;
if (!GetFullPathNameW(path, PATH_SIZE, buffer, lppPart)) {
fwprintf(stderr, L"warning: %d unable to expand path %s\n", GetLastError(), path);
return;
}
if (!AddDllDirectory(buffer)) {
DWORD error = GetLastError();
if (error != ERROR_FILE_NOT_FOUND) {
fwprintf(stderr, L"warning: %d unable to set DLL search path for %s\n", GetLastError(), path);
}
}
}
int locate_py_main(int argc, wchar_t **argv) {
/*
* We have to dynamically locate Python3.dll because we may be loading a
* Python native module while running. If that module is built with a
* different Python version, we will end up a DLL import error. To resolve
* this, we can either ship an embedded version of Python with us or
* dynamically look up existing Python distribution installed on user's
* machine. This way, we should be able to get a consistent version of
* Python3.dll and .pyd modules.
*/
HINSTANCE python_dll;
Py_Main pymain;
// last added directory has highest priority
add_search_path(L"C:\\Python36\\");
add_search_path(L"C:\\Python37\\");
add_search_path(L"..\\python\\");
python_dll = LoadLibraryExW(L"python3.dll", NULL, LOAD_LIBRARY_SEARCH_DEFAULT_DIRS);
if (python_dll != NULL) {
pymain = (Py_Main) GetProcAddress(python_dll, "Py_Main");
if (pymain != NULL) {
(pymain)(argc, argv);
} else {
fprintf(stderr, "error: %d unable to load Py_Main\n", GetLastError());
}
FreeLibrary(python_dll);
} else {
fprintf(stderr, "error: %d unable to locate python3.dll\n", GetLastError());
return 1;
}
return 0;
}
int wmain() { int wmain() {
/* /*
* This executable will be prepended to the start of a Python ZIP archive. * This executable will be prepended to the start of a Python ZIP archive.
...@@ -44,8 +104,6 @@ int wmain() { ...@@ -44,8 +104,6 @@ int wmain() {
* "-h" * "-h"
* } * }
*/ */
#define PATH_SIZE 1024
wchar_t full_path_to_argv0[PATH_SIZE]; wchar_t full_path_to_argv0[PATH_SIZE];
DWORD len = GetModuleFileNameW(NULL, full_path_to_argv0, PATH_SIZE); DWORD len = GetModuleFileNameW(NULL, full_path_to_argv0, PATH_SIZE);
if (len == 0 || if (len == 0 ||
...@@ -63,5 +121,5 @@ int wmain() { ...@@ -63,5 +121,5 @@ int wmain() {
pyargv[0] = full_path_to_argv0; pyargv[0] = full_path_to_argv0;
pyargv[1] = full_path_to_argv0; pyargv[1] = full_path_to_argv0;
return Py_Main(__argc + 1, pyargv); return locate_py_main(__argc + 1, pyargv);
} }
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment