1. Vladimir Esipov
  2. Valentina Server
  3. Freitag, Juli 19 2024, 01:14 PM
  4.  Abonnieren via E-Mail
Python 3.12 and Microsoft Visual C++ Redistributable package installed. Environment paths are ok.

pip install duckdb

I'm trying to create a function that accesses the DuckDB package


CREATE FUNCTION "GetVersion"() RETURNS String LANGUAGE valentina_python
AS
$$
import sys

if 'd:/pyt/lib/site-packages/' not in sys.path:
sys.path.append('d:/pyt/lib/site-packages/')

import duckdb

ver = duckdb.sql("SELECT version()")
return ver
$$;

SELECT GetVersion();

--> Kernel error: 0x50000. DLL load failed while importing duckdb: The specified module was not found.


Help me understand, how to import a DuckDB package correctly?
Kommentar
There are no comments made yet.
Sergey Pashkov Akzeptierte Antwort
Hello Vladimir,

The procedure looks correct, we'll check it.
Kommentar
There are no comments made yet.
Vladimir Esipov Akzeptierte Antwort
Actually, the problem is on line

import duckdb

By the way, the example you recommended on the WIKI page using the NUMPU package from the function is also not working. :)

import numpy as np

Returns:
numpy import error: You should not try to import numpy from the source directory;
please exit the numpy source tree and restart the Python interpreter from there.
Kommentar
There are no comments made yet.
Sergey Pashkov Akzeptierte Antwort
Unfortunately, it is a Windows-specific problem where binary packages depend on a Python DLL, and Python is built-in in VKernel, so the dependency is not satisfied. On other platforms, it works as expected.

Do you have any specific tasks for DuckDB in Valentina stored procedures (some integration?), or is it a workaround until we add DuckDB native support in Valentina Studio?
Anhänge
Kommentar
There are no comments made yet.
Vladimir Esipov Akzeptierte Antwort
Sergey, hello! Thanks for the answer...
To a greater extent - integration. If there is access to DuckDB from VDB, then there will be access to all the engines that DDB can attach + a sea of ​​other features that DDB can do.
For example, VDB cannot LOAD DATA INFILE from files (I asked Ruslan for this feature about 8 years ago), and it is also poorly implemented in Studio. Duck allows you to work wonders.
It would be ideal if the VDB engine could work transparently with DDB itself (but that's fantasy...).

Regarding compatibility, I first tried with Python 3.12.4, then 3.12.0. To no avail.
This is not even a matter of DuckDB - and, as I understand it, there will also be problems with other external packages. Yes, with Windows there is always some kind of ass coming out...

What version do you have in VKernel?
Kommentar
There are no comments made yet.
Sergey Pashkov Akzeptierte Antwort
Hello Vladimir,

Thank you for the clarification. Of course, we'll try to get it to work.

The version in VKernel is 3.11, but the problem is that even packages for 3.11 will not work (they search for a separate python311.dll).
Kommentar
There are no comments made yet.
Sergey Pashkov Akzeptierte Antwort

For example, VDB cannot LOAD DATA INFILE from files (I asked Ruslan for this feature about 8 years ago), and it is also poorly implemented in Studio. Duck allows you to work wonders.


So, it makes it possible to copy records from various sources via DuckDB to VDB tables, right?

LOAD DATA INFILE - is just the SQL command necessary? There is an API method for Import, but maybe some options are missing (e.g., replacing records with duplicated keys)?
Kommentar
There are no comments made yet.
Vladimir Esipov Akzeptierte Antwort
Yes, MariaDB LOAD DATA INFILE allows ignore or replace duplicates, as well as transform incoming data via SET col_name = expr,...

But DuckDB went much further
- reading directly and/or import-export of almost all formats (txt, JSON, parquet, SQLite...), from the file system or http(s), S3, Azure.
There is a built-in sniffer that will analyze the field types in the file and create the necessary table. You can read|write from|to custom partitions. You can specify a reject table where failed lines with a description of the error, etc. will be reset.
You can loaded the desired Extensions and, via ATTACH, read and write to various DBMSs (PostgreSQL, MySQL, ODBC...).

It will most likely not be possible to write directly from DuckDB via Python to VDB until the data types are forwarded, but I think you can always convert it to text and dump it into a temporary table.
If you integrate DuckDB, the need for LOAD DATA will disappear automatically.

Q: Are you working on ODBC (as Ruslan planned), or will the Studio support the DuckDB C++ API directly?

Если уже есть что потестить - я всегда пожалуйста, с превеликим удовольствием :D
Kommentar
There are no comments made yet.
Vladimir Esipov Akzeptierte Antwort
Кстати, о девушках - а вы не пробовали из Студии подключаться к ClickHouse - ведь он поддерживает MySQL-протокол.
Я пару лет назад писал в Lazarus прогу, которая по MySQL-протоколу читала из ClickHouse весьма шустро..
Тогда мне было не до Валентины, а сейчас, жаль, нет того сервака, иначе проверил бы. Аж самому стало интересно. Ведь CH тоже может взаимодействовать с целой кучей БД.
От CH - BigData, а у вас ReportServer. Данные покрутил и тут же на печать. Красота!
Проверьте!
Если всё ОК - можно будет заявить о поддержке Студией (частичной) ещё одной популярной аналитической СУБД!
Причём на-шару! :D
Kommentar
There are no comments made yet.
Sergey Pashkov Akzeptierte Antwort
C++ interface will be used directly.

We haven't tried Clickhouse, thank you, it's interesting.
Kommentar
There are no comments made yet.
  • Seite :
  • 1


There are no replies made for this post yet.
However, you are not allowed to reply to this post.

Categories

Announcements & News
  1. 0 subcategories
Valentina Studio
  1. 2 subcategories
Valentina Server
  1. 4 subcategories
Valentina Database ADK
  1. 0 subcategories
Valentina Reports ADK
  1. 0 subcategories
Other Discussions
  1. 2 subcategories
BETA Testing
  1. 0 subcategories
Education & Research
  1. 0 subcategories