I am loading data from an Oracle database with the Database connection Load processor. Some columns have types like DECIMAL(38,10) in Oracle.
I then save the data into a ONE DATA data table X using “Create or append”. The initial saving (create table) is working fine and the columns are cast as double.
However if I run the workflow again trying to append the data to the existing table I get the error:
Column COL_X can’t be casted from DecimalType(38,10) to DoubleType to match the schema of the existing Data Table.
However the append did work in the past when I loaded the already existing data table X, do a join with the new input data from Oracle (to check which data from Oracle is not present in the existing data table X), select only the Oracle columns in the join and append then the input from the Oracle DB to X.
What could be the reason that with an intermediate join the append is working, while it is failing when appending directly?
Furthermore, after some appends in the past, I am now not able to load the data table X anymore (sample or in a workflow) and get an error like:
Could not fetch Data Table by id
[ "Job aborted due to stage failure: Task 0 in stage 2129.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2129.0 (TID 27122, instance-cropped-by-me, executor 6): org.apache.spark.sql.execution.QueryExecutionException: Parquet column cannot be converted in file maprfs:///path-cropped-by-me.parquet/part-00006-7729607d-7fd5-44e8-88b4-86ac503f3097-c000.snappy.parquet. Column: [Col_X], Expected: decimal(38,10), Found: DOUBLE
What could be the reason to being not able to load this data, despite I never got errors when appending it?
cc: @Viola , @ali